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Overview 


CL. have shown that math skills in early childhood are uniquely and strongly predictive of later out- 
comes across a range of domains and well into adulthood, including the likelihood of graduating from 
high school and college completion. The Making Pre-K Count and High 5s studies were designed to rigor- 
ously test the short- and long-term effects of improving children’s math experiences in prekindergarten 
(pre-K) and kindergarten. 


Making Pre-K Count provided pre-K teachers in New York City with a high-quality, evidenced-based math 
curriculum (Building Blocks) and ongoing teacher training and coaching. The Making Pre-K Count study com- 
pared students who were exposed to this curriculum with their peers in pre-K as usual in public school and 
community-based sites. The High 5s math program was developed to offer children who had received Making 
Pre-K Count in pre-K in public schools hands-on, supplemental math enrichment in small groups, or clubs, 
outside of regular instructional time in kindergarten. The High 5s study compared students assigned to Making 
Pre-K Count in pre-K and High 5s in kindergarten with children assigned to Making Pre-K Count in pre-K and 
kindergarten as usual. The studies also compared two years of math enrichment with no math enrichment. 


The studies used random assignment and tracked children through third grade to test the effects of these math 
enrichment programs. The confirmatory outcome examined was children’s third-grade math scores. 


KEY FINDINGS 


e Making Pre-K Count: Though not statistically significant, Making Pre-K Count had small, positive, longer- 
term impacts on children’s third-grade math test scores, compared with pre-K as usual in public school and 
community-based sites. 


e High 5s: The impact of High 5s on children’s third-grade math test scores in public schools, over and above 
the effect of Making Pre-K Count alone, was close to zero and not statistically significant. 


e Making Pre-K Count plus High 5s: Making Pre-K Count and High 5s together had moderate, statistically 
significant impacts on children’s math test scores, compared with pre-K and kindergarten as usual in public 
schools. 


The study team also explored the impact of these two math interventions on children’s third-grade literacy test 
scores, chronic absenteeism, retention in a grade, and placement in special education. These exploratory 
analyses suggest that Making Pre-K Count alone and the two years of math enrichment together reduced 
chronic absenteeism and improved children’s literacy test scores, though findings were not always statistically 
significant for literacy test scores. 


Taken together, the Making Pre-K Count and High 5s studies present new evidence about the long-term effects 
of early math interventions on children’s later outcomes. Early math enrichment experiences can lead to lasting 
gains for children across a variety of outcome domains, even years later. The findings suggest that high-quality 
early math instructional practices could make a difference, particularly for children with the greatest need. 
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Executive Summary 


tudies have found that math skills in early childhood are uniquely and strongly associated with 
ee later in life. Strong early math skills are correlated with not only later math achieve- 
ment, but also with better reading skills and executive functioning.’ Further, studies have shown 
that early math competencies predict outcomes well into adulthood, including the likelihood of 
graduating from high school and college completion.” The Making Pre-K Count and High 5s stud- 
ies were designed to test the impact of early math enrichment interventions on children’s short- 
and longer-term outcomes. 


The Making Pre-K Count study was designed to rigorously assess the short- and long-term effects of 
improving children’s math experiences in prekindergarten (pre-K). Making Pre-K Count operated in 
community-based and public school pre-K classrooms in New York City that served mostly children 
from families with low incomes. Making Pre-K Count provided teachers with a high-quality math cur- 
riculum (Building Blocks) and ongoing teacher training and coaching.’ In the Making Pre-K Count 
study, whole pre-K sites—community-based organizations and public schools—were randomly as- 
signed to receive either the evidence-based math curriculum plus coaching and training (n = 35) or 
continue with pre-K-as-usual (n= 34). During the time when the program was implemented, there was 
a growing emphasis on early math instruction in all New York City schools.* Children in the control 
group therefore received more math instruction than had previously been typical in prior studies of 


early math education programs.” 


The High 5s program was developed to offer supplemental math enrichment outside of regular in- 
structional time to kindergarten children who had received Making Pre-K Countin pre-K. High 5s 
grouped three to four children with one facilitator for math clubs that met three times a week for 30 
minutes each session, outside of regular classroom instruction. Children who were in public schools 
that implemented Making Pre-K Count and stayed in the same public school were eligible for High 
5s. In those Making Pre-K Count program public schools, individual children were randomly as- 
signed within a school to either two years of math enrichment (Making Pre-K Countin pre-K plus 


'Greg J. Duncan, Chantelle J. Dowsett, Amy Claessens, Katherine Magnuson, Aletha C. Huston, Pamela Klebanov, Linda S. 
Pagani, Leon Feinstein, Mimi Engel, and Jeanne Brooks-Gunn, “School Readiness and Later Achievement,” Developmental 
Psychology 43, 6 (2007): 1,428-1,446; Douglas H. Clements, Julie Sarama, and Carrie Germeroth, “Learning Executive Func- 
tion and Early Mathematics: Directions of Casual Relations,” Early Childhood Research Quarterly 36 (2016): 79-90. 

*Greg J. Duncan and Katherine Magnuson, “Investing in Preschool Programs,” The Journal of Economic Perspectives 27, 2 
(2013): 109-132; Greg J. Duncan and Katherine Magnuson, “The Nature and Impact of Early Achievement Skills, Attention 
Skills, and Behavior Problems,” pages 47-69 in Greg J. Duncan and Richard J. Murnane (eds.), Whither Opportunity: Ris- 
ing Inequality, Schools, and Children’s Life Chances (New York: Russell Sage, 2011). 

8Douglas H. Clements and Julie Sarama, Building Blocks: Teacher's Edition (Columbus, OH: McGraw-Hill Companies, Inc., 
2013) 

4Pamela A. Morris, Shira K. Mattera, and Michelle F. Maier, Making Pre-K Count: Improving Math Instruction in New York 
City (New York: MDRC, 2016). 

Julie Sarama, Douglas H. Clements, Prentice Starkey, Alice Klein, and Ann Wakeley, “Scaling Up the Implementation of a 
Pre-Kindergarten Mathematics Curriculum: Teaching for Understanding with Trajectories and Technologies,” Journal of 
Research on Educational Effectiveness 1, 2 (2008): 89-119; Douglas H. Clements, Julie Sarama, Mary Elaine Spitler, Alissa 
A. Lange, and Christopher B. Wolfe, “Mathematics Learned by Young Children in an Intervention Based on Learning Tra- 
jectories: A Large-Scale Cluster Randomized Trial,” Journal for Research in Mathematics Education 42, 2 (2011): 127-166. 


ES-1 


High 5s in kindergarten, n = 320) or one year of math enrichment (Making Pre-K Count in pre-K 
and kindergarten as usual, n = 335). 


The studies were developed as part of the Robin Hood Early Childhood Research Initiative, which 
was established toidentify and rigorously test promising early childhoodinterventions. The initiative 
is a partnership between Robin Hood, one of New York City’s leading antipoverty organizations, and 
MDRC, a nonprofit, nonpartisan education and social policy research organization. Its flagship pro- 
jects, Making Pre-K Count and High 5s, were conducted in collaboration with Bank Street College of 
Education and RTI International and supported with lead funding from the Heising-Simons Foun- 
dation, the Overdeck Family Foundation, and the Richard W. Goldman Family Foundation. This 
report is the fifth report based on these studies. 


A key feature of the Making Pre-K Count and High 5s studies was a focus on developing the math 
competencies of children enrolled in pre-K as a pathway to improving a broader set of children’s 
outcomes into elementary school. Third grade is considered a particularly important moment in a 
child’s educational experience. Literacy skill levels in third grade predict rates of high school com- 
pletion.® While third grade may bea critical time for ensuring children’s future success, few studies 
have tracked the effects of pre-K programs in the longer term, and the evidence on whether gains 
from pre-K interventions are sustained into early elementary school and beyond from those thathave 
is mixed.’ 


The design of the Making Pre-K Count and High 5s studies makes it possible to rigorously assess 
the impact on children’s outcomes from one year of math enrichment in pre-K (Making Pre-K 
Count compared with pre-K as usual), an additional year of math enrichment in kindergarten 
(Making Pre-K Count plus High 5s in kindergarten compared with Making Pre-K Count only), 
and two years of math enrichment (Making Pre-K Count plus High 5s in kindergarten compared 
with pre-K and kindergarten as usual). The samples of sites and children used in these analyses do 
not perfectly overlap, therefore the findings cannot be directly compared with one another. How- 
ever, considered together, these analyses provide useful insights about the longer-term effects of 
early math enrichment interventions. 


Earlier reports on these studies examined the effects of math enrichmentat the end of pre-K and at 
the end of kindergarten.* The pre-K math program had small but not statistically significant effects 
on children’s math skills by the end of kindergarten, and statistically significant effects on children’s 


SDuncan and Magnuson (2011); Catherine E. Snow, Susan M. Burns, and Peg Griffin, Preventing Reading Difficulties in 
Young Children (Washington, DC: National Academy Press, 1998); Donald J. Hernandez, Double Jeopardy: How Third- 
Grade Reading Skills and Poverty Influence High School Graduation (Baltimore, MD: Annie E. Casey Foundation, 2011). 
7Janet Currie and Duncan Thomas, “Does Head Start Make a Difference?” The American Economic Review 85, 3 (1995): 
341-364; Eliana Garces, Duncan Thomas, and Janet Currie, “Longer-Term Effects of Head Start,” The American Economic 
Review 92, 4 (2002): 999-1,012; James J. Heckman, Jora Stixrud, and Sergio Urzua, “The Effects of Cognitive and Non- 
cognitive Abilities on Labor Market Outcomes and Social Behavior,” Journal of Labor Economics 24, 3 (2006): 411-482; 
Jens Ludwig and Douglas L. Miller, “Does Head Start Improve Children’s Life Chances? Evidence from a Regression Dis- 
continuity Design,” The Quarterly Journal of Economics 122, 1 (2007): 159-208; David Deming, “Early Childhood Interven- 
tion and Life-Cycle Skill Development: Evidence from Head Start,” American Economic Journal: Applied Economics 1, 3 
(2009): 111-134; Lawrence J. Schweinhart, “Long-Term Follow-Up of a Preschool Experiment,” Journal of Experimental 
Criminology 9, 4 (2013): 389-409. 

8Morris, Mattera, and Maier (2016); Shira K. Mattera, Robin Jacob, and Pamela A. Morris, Strengthening Children’s Math 
Skills with Enhanced Instruction: The Impacts of Making Pre-K Count and High 5s on Kindergarten Outcomes (New York: 
MDRC, 2018); 
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math attitudes and working memory. The kindergarten math clubs had positive effects equivalent to 
an additional 2.5 months of math learning on one of two math measures at the end of kindergarten. 
The two programs jointly had a positive effect on one of two measures of children’s math skills by 
the end of kindergarten, equivalent to over four months of additional math learning. 


The current report presents the longer-term impacts on third-grade outcomes. The confirmatory 
outcome for these studies is children’s third-grade math scores, since math skills are the direct target 
of the Making Pre-K Count and High 5s programs. The key confirmatory findings at the end of third 
grade are the following: 


¢ One year of math enrichment in pre-K: Though notstatistically significant, Making Pre-K Count 
had a small, positive, longer-term impact on children’s third-grade math test scores (ES = 0.10), 
compared with pre-K as usual in control sites. 


e An additional year of math enrichment in kindergarten: The impact of High 5s on children’s 
third-grade math test scores in public schools, over and above the effect of Making Pre-K Count 
alone, was close to zero and not statistically significant (ES = 0.02). 


e Two years of math enrichment (pre-K and kindergarten): Making Pre-K Count and High 5s 
together had moderate, statistically significant impacts on children’s math test scores, compared 
with pre-K and kindergarten as usual in public schools (ES = 0.34). 


The finding that two years of math enrichment (Making Pre-K Count plus High 5s) had moderate 
effects seems counter-intuitive given the small effects of each of the two interventions separately. 
This pattern of results is likely due to differences among the samples of children used in each analysis. 
Exploratory subgroup analyses suggest that early math enrichment may have been particularly ben- 
eficial for children with the most room to grow. Making Pre-K Count’s impacts on third-grade math 
scores were fairly large—ranging from one-quarter to over a third of a standard deviation—for those 
children entering pre-K with the weakest language and attention skills. It appears that children with 
the lowest scores on the third-grade tests were more prevalent in the sample used to estimate the 
impact of two years of early math enrichment, and this difference may have contributed to the larger 
impacts observed in the sample. 


The Making Pre-K Count and High 5s studies were also designed to test whether early math enrich- 
ment could have effects on outcomes beyond math skills. These outcomes are not the explicit focus 
of the programs, and empirical evidence that early math programming can have an impact on these 
outcomes is more limited. 


Exploration of these outcomes suggest that Making Pre-K Count alone and when supplemented with 
High 5s may reduce chronic absenteeism and improve children’s literacy test scores in third grade, 
though the findings are not always statistically significant. Making Pre-K Count alone, and when 
combined with an additional year of early math enrichment, led toa statistically significant reduction 
in children’s chronic absenteeism in third grade, equivalent to about 9 percentage points or 28 per- 
cent. The effects of the programs on children’s third-grade literacy test scores were similar in magni- 
tude to the effects on third-grade math scores. None of the early math enrichment programs had an 
effect, positive or negative, on children’s retention in a grade or placement in special education. 
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IMPLICATIONS 


The Making Pre-K Count and High 5s studies rigorously tested the potential of early math enrich- 
ment interventions to both improve children’s short-term outcomes and sustain these effects into 
elementary school. 


e These findings contribute to growing evidence about the longer-term importance of high- 
quality early math instruction for children, particularly those with the most room to grow. 


Correlational studies have suggested that early math skills could be a powerful lever for improving 
children’s later skills, in math and in other domains. These studies hypothesize that early math learn- 
ing may help children develop other skills, suchas language skills and executive functioning, which 
may set the stage for effects on a wider range of longer-term outcomes. However, few studies have 
examined the long-term effects of enriched early math instruction to see whether or not the gains are 
sustained into elementary school. 


The Making Pre-K Count and High 5s studies were designed not only to test the effects of the pro- 
grams on math skills, but also to test whether early math programs could affect outcomes in other 
domains as well. The Making Pre-K Countand High 5s studies add to the base of evidence by demon- 
strating that enriched early math instruction has the potential to improve children’s skills, both in 
math and other domains, and to sustain those improvements for at least four years. 


Prior findings indicate that Making Pre-K Count had small, positive effects on outcomes in pre-K 
and kindergarten across multiple domains, including math skills, executive functioning, and chil- 
dren’s attitudestoward math. This report finds thatthe effects of Making Pre-K Countwere sustained 
into third grade, with small effects on children’s math and literacy scores and favorable effects on 
chronic absenteeism. The effects of Making Pre-K Count on math test scores are comparable to those 
of other similar curricula implemented at scale and translate to approximately 12 percent of the 
achievement gap in fourth grade between low-income children and their high-income peers.? When 
children received two years of early math enrichment, the effects on tests are equivalent to approxi- 
mately 40 percent of the achievement gap in fourth grade between low-income children and their 
high-income peers. 


8Long-term effects from other interventions implemented at scale range from effects of 0.28 on third-grade literacy test 
scores from a social-emotional learning intervention to effects of 0.26 on fifth-grade math skills in a study of Building 
Blocks. Meghan P. McCormick, Robin Neuhaus, Erin E. O’Connor, Hope |. White, E. Parham Horn, Samantha Harding, 
Elise Cappella, and Sandee McClowry, “Long-Term Effects of Social-Emotional Learning on Academic Skills: Evidence 
from a Randomized Trial of INSIGHTS,” Journal of Research on Educational Effectiveness 14, 1 (2021): 1-27; Tyler W. 
Watts, Greg J. Duncan, Douglas H. Clements, and Julie Sarama, “What Is the Long-Run Impact of Learning Mathematics 
During Preschool?” Child Development 89, 2 (2018): 539-555. Effect sizes in this study are standardized measures of the 
difference in outcomes at the end of third grade for the control and program groups. To contextualize these impacts, effect 
sizes are compared with other available standardized data on the difference in achievement between children who are eli- 
gible for free or reduced price lunch and those who are not eligible. Using National Assessment of Educational Progress 
data from 2,000 for children at the end of fourth grade, the achievement gap between those eligible for free or reduced 
price lunch and those not eligible was equivalent to 0.85 standardized units. Carolyn J. Hill, Howard S. Bloom, Alison Re- 
beck Black, and Mark W. Lipsey, “Empirical Benchmarks for Interpreting Effect Sizes in Research,” Child Development 
Perspectives 2, 3 (2008): 172-177. The effect of Making Pre-K Count on third-grade math scores (0.10) is equivalent to 12 
percent of that difference. 
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The effects on chronic absenteeism are substantively meaningful. Rates of chronic absenteeism were 
approximately 33 percent among third-graders in the control group and 24 percent in the program 
group. Reducing absenteeism by 9 percentage points for third-graders citywide in New York City 
could lead to over 7,000 fewer chronically absent third-graders per year.'® Chronic absenteeism is 
associated with lower achievement in reading and math and poor socioemotional outcomes, even 
after controlling for a wide range of background characteristics." 


The pattern of long-term effects, which suggests that impacts were largest for those with the most 
room to grow, supports the “academic risk hypothesis,” which posits that effects of early childhood 
education may be the largest for children who need the most support.” 


e Well-designed math enrichment programs can have an effect even when layered on top of ex- 
isting math instruction. 


The Making Pre-K Count program compared students who were exposed toa well-implemented, ev- 
idence-based early math enrichment program with their peers in other New York City pre-K pro- 
grams. All students in the sample attended pre-K. During the time in which the program was imple- 
mented, there was a growing emphasis on early math instruction in New York City schools, and even 
children in the control group received more math instruction than had been typical in previous stud- 
ies of early math enrichment interventions. '* Thus, these long-term impacts reflect the added value 
of implementing high-quality math instruction in pre-K, above and beyond the impactof pre-K itself 
and of typical pre-K math instruction. 


The Making Pre-K Countand High 5s studies contribute newevidence aboutthe effects ofearly math 
enrichment experiences on children’s later outcomes. Such experiences can lead to lasting gains for 
children, particularly for children with the greatest need. 


New York City had 78,141 third-graders in 2019-2020. New York State Education Department, “NYC Public Schools at a 
Glace 2019-20” (2020), website: www.data.nysed.gov/profile. php ?instid= 7889678368. An analysis by New York University 
estimated that 22.8 percent of students were chronically absent in 2018. Research Alliance for New York City Schools, 
“How Has Attendance in NYC Schools Changed Over Time?” (2019), website: www.steinhardt.nyu.edu/research-alli- 
ance/research/spotlight-nyc-schools/how-has-attendance-nyc-schools-changed-over-time. According to those numbers, 
an estimated 17,816 third-graders would be chronically absent. After a reduction of chronic absenteeism by 9 percentage 
points, an estimated 10,784 third-graders would be chronically absent. 

"'Mariajosé Romero and Young-Sun Lee, A National Portrait of Chronic Absenteeism in the Early Grades (New York: Na- 
tional Center for Children in Poverty, 2007); Michael A. Gottfried, “Chronic Absenteeism and Its Effects on Students’ Aca- 
demic and Socioemotional Outcomes,” Journal of Education for Students Placed at Risk 19, 2 (2014): 53-75. 

"Bridget K. Hamre and Robert C. Pianta, “Can Instructional and Emotional Support in the First-Grade Classroom Make a 
Difference for Children at Risk of School Failure?” Child Development 76, 5 (2005): 949-967; Bridget K. Hamre and Robert 
C. Pianta, “Early Teacher-Child Relationships and the Trajectory of Children’s School Outcomes through Eighth Grade” 
Child Development 72, 2 (2001): 625-638. 

Morris, Mattera, and Maier (2016); Sarama et al. (2008); Clements et al. (2011). 
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Introduction 


ee have found that math skills in early childhood are uniquely and strongly predictive of out- 
comes later in life. Strong early math skills are associated with not only later math achievement, 
but also better reading skills and executive functioning.’ Further, studies have shown that early math 
competencies predict outcomes well into adulthood including the likelihood of graduating from high 
school and college completion.” While compelling, these studies are all based on correlational data, 
and few studies to date have tried to rigorously assess the impact of improving early math skills on 
later outcomes. 


The Making Pre-K Countand High 5s studies were designed to rigorously assess the short- and long- 
term effects of improving children’s math experiences in prekindergarten (pre-K) and kindergarten. 
Making Pre-K Count began in fall 2013 and provided pre-K teachers in New York City with a high- 
quality math curriculum (Building Blocks) and ongoing teacher training and coaching.’ The Making 
Pre-K Count study compared students who were exposed to a well-implemented, evidence-based 
math program with their peers in other New York City pre-Ks.* Making Pre-K Count operated in 
community-based and public school pre-Ks that served mostly children from families with low in- 
comes. During the time when the program was implemented, there was a growing emphasis on early 
math instruction in New York City schools, and children in the control group received more math 
instruction than had been observed in prior studies of early math education programs.° 


The High 5s program was developed to offer supplemental math enrichment outside of regular in- 
structional time to kindergarten children who had received Making Pre-K Countin pre-K. High 5s 
grouped three to four children with one facilitator for math clubs that met three times a week for 30 
minutes eachsession, outside ofregularclassroom instruction. The High 5s study was only conducted 
in the public school sites, where children could stay in the same school for pre-K and kindergarten. 
The community-based Pre-K sites were not included in the High 5s study because children who at- 
tended them dispersed to schools across the city for kindergarten. The High 5s study compared chil- 
dren who were offered Making Pre-K Count in pre-K and High 5s inkindergarten with children who 
were offered only Making Pre-K Countin pre-K. 


‘Duncan et al. (2007); Clements, Sarama, and Germeroth (2016). 

*Duncan and Magnuson (2013); Duncan and Magnuson (2011). 

8Clements and Sarama (2013). 

4Morris, Mattera, and Maier (2016). 

5Morris, Mattera, and Maier (2016); Sarama et al., (2008); Clements et al. (2011). 


The design of the Making Pre-K Count and High 5s studies also makes it possible to evaluate the 
effect of two years of early math enrichment (Making Pre-K Count plus High 5s) compared with no 
early math enrichment. The analyses comparing these impacts are based on samples used in both 
Making Pre-K Count and High 5s studies and therefore only include children eligible for High 5s— 
that is, children in public schools sites who stayed in the same school for pre-K and kindergarten. 


Previous reports examined the impact of Making Pre-K Count and High 5s on children’s outcomes 
at the end of both pre-K and kindergarten.° By the end of kindergarten, Making Pre-K Count had 
small, positive, but not consistently statistically significant impacts on one of two measures of chil- 
dren’s math skills, and statistically significant impacts on both attitudes toward math and working 
memory skills, compared with children who had not received math enrichment in pre-K. Making 
Pre-K Countdid nothave statistically significant impacts on children’s language or inhibitory control 
skills. High 5s led to positive and statistically significant impacts on one of two measures of students’ 
math skills, when compared with Making Pre-K Count alone. High 5s did not have statistically sig- 
nificant impacts on children’s attitudes toward math, language skills, or executive functioning, when 
compared with students who received Making Pre-K Count only. The two years of aligned math 
enrichment (Making Pre-K Count plus High 5s) led to positive and statistically significant impacts 
on one of two measures of students’ math skills and also led to more positive attitudes toward math 
among students, compared with those who had received no math enrichment in either pre-K or kin- 
dergarten. The two years of combined math enrichment programming did not have statistically sig- 
nificant impacts on children’s language skills or executive functioning, or on the other, more global, 
measure of math skills. 


The studies were developed as part of the Robin Hood Early Childhood Research Initiative, which 
was established to identify and rigorously test promising early childhood interventions. That initia- 
tive is a partnership between Robin Hood, one of New York City’s leading antipoverty organizations, 
and MDRC, a nonprofit, nonpartisan education and social policy research organization. Its flagship 
projects, Making Pre-K Countand High 5s, were conductedin collaboration with Bank Street College 
of Education and RTI International and supported with lead funding from the Heising-Simons 
Foundation, the Overdeck Family Foundation, and the Richard W. Goldman Family Foundation. 


This report presents longer-term effects of the two interventions on children’s outcomes in third 
grade. It is the fifth report based on these studies. 


WHY THIRD GRADE? 


A key feature of the Making Pre-K Count and High 5s studies was a focus on developing pre-K 
children’s math competencies as a pathway to improving a broader set of children’s outcomes into 
elementary school. Third grade is considered a particularly important moment in a child’s educa- 
tional experience. Research has consistently found that third-grade reading outcomes strongly pre- 
dict future academic challenges including dropping out of high school.’ Similarly, strong and 


6Morris, Mattera, and Maier (2016); Mattera, Jacob, and Morris (2018). 
7Snow, Burns, and Griffin (1998); Hernandez (2011). 


sustained math skills in elementary school predict higher rates of high school completion and col- 
lege enrollment.® 


Third grade may be a critical time for putting children on track for future success, but existing evi- 
dence on whether pre-K interventions are able to sustain any early gains into elementary school is 
mixed. While initial impacts on cognitive and achievement test scores tend to fade, some of these 
early childhood education programs nonetheless appear to have important long-term effects on high 
school completion, college attendance, earnings, healthy behaviors, and criminal involvement.’ 


Although there are few studies with which to test the hypothesis, Heckman, Stixrud, and Urzua did 
posit in a 2006 study that some of the longer-term effects of pre-K programs are a result of impacts 
on a set of “non-cognitive skills” that are frequently unmeasured, such as executive functioning and 
self-regulation, academic motivation or attitudes, and social-emotional skills.’ Correlational find- 
ings suggest that math skills may have spillover effects into these non-cognitive domains, which may 
help to sustain longer-term impacts.'' The Making Pre-K Count and High 5s studies were designed 
to examine this hypothesis by testing the short- and long-term effects of early math enrichment 
across both cognitive and non-cognitive domains. 


Another hypothesis for the observed fading out of the effects on cognitive and achievement outcomes 
is that the instruction that children receive in pre-K, in terms of the instructional content or peda- 
gogical approach, is not well aligned with the instruction they receive in kindergarten and be- 
yond.” This “sustaining environments” hypothesis suggests that better aligning instructional expe- 
riences in pre-K with those in early elementary school could help sustain the impacts of programs 
implemented in pre-K.'’ The High 5s study was expressly designed to test whether an additional year 
of aligned math enrichment would help maintain the effects of early math enrichment into elemen- 
tary school. 


THIRD-GRADE FOLLOW-UP 


This report presents the longer-term impacts of early math enrichment in pre-K (Making Pre-K 
Count) and in kindergarten (High 5s) on children’s third-grade outcomes. The research team ob- 
tained data on children’s third-grade test scores, chronic absenteeism, retention ina grade, and place- 
mentin special education from the New York City Department of Education administrative records. 
The confirmatory outcome for these studies is children’s third-grade math scores, since math skills 
are the direct target of the Making Pre-K Count and High 5s programs. The Making Pre-K Count 
and High 5s studies were also designed to test whether early math enrichment could have effects on 
outcomes beyond math skills. These other outcomes are considered exploratory because they are not 


8Duncan and Magnuson (2011). 

8Currie and Thomas (1995); Garces, Thomas, and Currie (2002); Heckman, Stixrud, and Urzua (2006); Ludwig and Miller 
(2007); Deming (2009); Schweinhart (2013). 

tHeckman, Stixrud, and Urzua (2006). 

"'Sarama, Lange, Clements, and Wolfe (2012); Blair, Knipe, and Gamson (2008). 

Engel, Claessens, and Finch (2013); Engel, Claessens, Watts, and Farkas (2015); Lee and Loeb (1995); Bailey, Jenkins, 
and Alvarez-Vargas (2020). 

Bailey, Jenkins, and Alvarez-Vargas (2020). 


the explicit focus ofthe programs, and empirical evidence that early math programming can have an 
impact on these outcomes is more limited. 


Though not statistically significant, Making Pre-K Count had a small, positive, longer-term impact 
on the studies’ confirmatory outcome of third-grade math test scores, compared with the pre-K as 
usual in the control sites. Making Pre-K Count led to small, positive, not statistically significant im- 
pacts on third-grade literacy test scores and moderate, statistically significant reductions in rates of 
chronic absenteeism, both exploratory outcomes. The program did not have effects on children’s 
retention in a grade or placement in special education. 


While High 5s had effects on children’s math skills in the year it was implemented, at the end of third 
grade, its impact on children’s math test scores, over and above the effect of Making Pre-K Count 
alone, was close to zero and not statistically significant. High 5s was implemented in public schools 
only. The effects of High 5s on exploratory outcomes in third grade were also close to zero and not 
statistically significant. 


At the end of third grade, Making Pre-K Count and High 5s together had moderate, statistically sig- 
nificant impacts on children’s math test scores, compared with pre-K and kindergarten as usual. 
These two years of aligned early math enrichmentalsoled to positive effects on children’s third-grade 
literacy test scores and chronic absenteeism, both exploratory outcomes. They did not have effects 
on children’s retention in a grade or placement in special education. This analysis only includes pub- 
lic schools and those students who remained in the same school for pre-K and kindergarten. 


The finding that two years of early math enrichment (Making Pre-K Count plus High 5s) had mod- 
erate effects seems counter-intuitive given the small effects of each of the two interventions sepa- 
rately. This pattern of results is likely due to differences among the samples of children used in each 
analysis. Exploratory subgroup analyses suggest that early math enrichment may have been particu- 
larly beneficial for children with the most room to grow. For example, Making Pre-K Count’s impacts 
on third-grade math scores were fairly large—ranging from one-quarter to over a third ofa standard 
deviation—for those children entering pre-K with the weakest language and attention skills. It ap- 
pears that children in the control group sample used to estimate the impact of two years of earlymath 
enrichment also had room to grow (having low third-grade test scores), and this difference may have 
contributed to the larger impacts observed in the sample. 


This report explains the above findings in greater detail. Chapter 2 presents the research design, sam- 
ple, and measures used in the studies. Chapter 3 describes the impacts of enhanced math experiences 
in pre-K and kindergarten on third-grade outcomes. Chapter 4 concludes with a discussion of the 
potential implications of these findings. 


Design, Sample, and Measures 


his chapter describes the design of the Making Pre-K Count and High $s studies and the analysis 
a bee impacts on third-grade outcomes. The studies rigorously tested the effects of early math en- 
richment in prekindergarten (pre-K) and kindergarten using randomized controlled trials. Children 
were tracked from pre-K through third grade, and data on their outcomes were collected from the 
New York City Department of Education administrative records. The studies examine the effects 
of (1) one year of math enrichment in pre-K (Making Pre-K Count), compared with pre-K as usual, 
(2) a supplemental year of math enrichmentin kindergarten (Making Pre-K Count plus High 5s), 
compared with math enrichment in pre-K only (Making Pre-K Count), and (3) two years of math 
enrichment (Making Pre-K Count plus High 5s), compared with pre-K and kindergarten as usual. 


DESIGN 


The research team tested Making Pre-K Count and High 5s usinga rigorous two-stage random as- 
signment design. Figure 2.1 illustrates this design. 


The Making Pre-K Count study tested the effects of an evidence-based pre-K math curriculum 
(Building Blocks), which was supported by two years of teacher trainingand in-classroom coaching.' 
In the study, the research team randomly assigned whole pre-K sites across New York City either to 
receive the evidence-based math curriculum and teacher training and coaching or to continue with 
pre-K as usual. The team blocked the sites by location (city borough), type (public school or commu- 
nity-based organization), and racial/ethnic composition (sites serving over 60 percent Hispanic chil- 
dren or sites serving children from other racial or ethnic backgrounds). * Groups of four to five sites 
were randomly assigned within blocks. The team estimated the effects of Making Pre-K Count by 
comparing the outcomes of children in the pre-K sites that implemented Making Pre-K Count with 
those of children in sites that continued with Pre-K as usual. 


‘Clements and Sarama (2013). 

*Morris, Mattera, and Maier (2016); Mattera, Jacob, and Morris (2018). Sites were “blocked” into groups of four to five be- 
fore randomization based on their borough, venue (Community-based organizations versus school-based sites), and the 
racial/ethnic composition of the children (whether the sites served primarily Hispanic children or not). Blocking achieves 
two goals: First, it reduces the risk of a poor match between program and control groups by accident given the small num- 
ber of units at the level of randomization; second, blocking in groups rather than pairs protects against the loss of sample 
sites between randomization and the study of program impact by allowing for the retention of all remaining sites if a single 
site drops out of the study. 
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Figure 2.1 
Making Pre-K Count (MPC) and High 5s Study Design 
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The High 5s program was implemented in the year after children completed pre-K. Children who 
attended public schools that received Making Pre-K Count and who stayed in the same school for 
pre-K and kindergarten were eligible for High 5s. For the High 5s study, the research team randomly 
assigned individual eligible children within their public school to either the High 5s program group 
(Making Pre-K Count plus High 5s) or a kindergarten-as-usual group (Making Pre-K Count only 
group). Children who attended pre-K and kindergarten in the same pre-K-as-usual public school 
sites, which did notimplementany early math enrichment, constituted the pre-K-and-kindergarten- 
as-usual control group. 


This two-stage sequential random assignment design thus created three experimental groups at the 
Making Pre-K Count public school sites. The research team used these groups to investigate two 
additional comparisons: (1) the effects of two years of early math enrichment (Making Pre-K Count 
plus High 5s), compared with one year of early math enrichment (Making Pre-K Count) and (2) the 
effects of two years of early math enrichment (Making Pre-K Count plus High 5s), compared with 
no math enrichment (pre-K and kindergarten as usual). Table 2.1 summarizes the analytic samples 
for the three confirmatory study comparisons. 


Table 2.1 


Analytic Samples for Third-Grade Confirmatory Study Comparisons 


MPC Plus High 5s vs. MPC Plus High 5s vs. 
Making Pre-K Count (MPC) MPC and Kindergarten Pre-K and Kindergarten 
vs. Pre-K as Usual as Usual as Usual 
Program Control Program Control Program Control 
Analytic Sample Group Group Group Group Group Group 
11 10 
24 22 
274 313 


NOTES: The second and third comparisons only include public schools and their students (no community-based 
organizations). 


The program students in the second and third comparisons are the same (n = 274). 


aThese sample sizes refer to the analytic samples in the study. They are inclusive of any students with outcome data 


in third grade. 


SAMPLE AND ANALYTIC STRATEGY 


Making Pre-K Count 


The Making Pre-K Count study was conducted in 69 pre-K sites, including 173 classrooms, across 
New York City. Thirty-five sites were randomly assigned to the program group and received the 
Building Blocks curriculum and teacher training and coaching. Thirty-four sites were randomly as- 
signed to the control group and continued with their usual pre-K practices. The sites in the program 
group implemented Making Pre-K Count over two school years, the 2013-2014 academic year and 
the 2014-2015 academic year. The first year was a “soft start” and allowed teachers to become familiar 


with the curriculum and receive training. For this reason, children participating in the program in 
the second academic year have been the main focus of the Making Pre-K Count study to date and 
make up the confirmatory sample (full implementation year sample) for this analysis. The research 
team also estimated the impacts of Making Pre-K Count on the outcomes of the students in the soft 
start year sample and three other exploratory samples to check that the pattern of effects was con- 
sistent across different samples. Appendix A describes these exploratory samples in greater detail. 


The analytic strategy for estimating the impacts of Making Pre-K Count on children’s third-grade 
outcomes builds on the strategy for estimating program impacts in kindergarten. The research 
team used multilevel modeling to account for the data’s nested structure, with children nested 
within pre-K sites and the sites nested within blocks. The team estimated the program’s impacts 
by comparing mean outcomes of students in the Making Pre-K Count group with those of students 
in the pre-K-as-usual control group, applying a regression adjustment for selected background 
characteristics and dummyvariables for random assignment blocks. See Appendix B for further 
details about the analysis. 


High 5s 


The High 5s study was embedded in the larger Making Pre-K Count study. It was conducted in the 
2015-2016 academic year in the 24 public schools that implemented Making Pre-K Count in 2013- 
2015. Children who stayed in the same public school for pre-K and kindergarten were eligible for 
High 5s. The research team randomly assigned the eligible children individually within their school 
to either a program group that received High 5s or a control group that received kindergarten as 
usual. The team randomly assigned a total of 655 children, 320 to the Making Pre-K Count plus High 
58 program group and 335 to the Making Pre-K Count only (kindergarten-as-usual) control group. 
These students make up the High 5s sample, which the research team used to estimate the effect of 
math enrichment in kindergarten over and above the impact of Making Pre-K Count alone. Because 
the High 5s study involved two stages of random assignment (one for the pre-K sites and the other 
for the individual kindergarteners), the Making Pre-K Count plus High 5s group could also be com- 
pared with a third group of students: those in public schools who received no math enrichment in 
either pre-K or kindergarten (the pre-K-and-kindergarten-as-usual control group). This two years 
of math sample thus consisted of the 320 students in the Making Pre-K Count plus High 5s program 
group and the 345 students in the pre-K-and-kindergarten-as-usual control group. 


The analytic strategy for estimating the impacts of High 5s on children’s third-grade outcomes also 
builds on the strategy used to estimate the program’s impacts in kindergarten. The research team 
estimated the effect of High 5s (over and above the effect of Making Pre-K Count only) by comparing 
the outcomes of children in the Making Pre-K Count plus High 5s group with the outcomes of chil- 
dren in the Making Pre-K Count-only group, applying a regression adjustment for selected back- 
ground characteristics and dummy variables for school. This analysis only included the 47 public 
school sites in the larger Making Pre-K Count study. See Appendix B for further details about the 
analysis. 


The research team estimated the effects of two years of early math enrichment by comparing the 
mean outcomes of children in the Making Pre-K Count plus High 5s group with the mean outcomes 


of children in the pre-K-and-kindergarten-as-usual control group. Because this comparison builds 
off the study’s cluster-level random assignment design, the team used multilevel modeling to account 
for the nested structure of the data. The team applied a regression adjustment to the analysis for 
selected background characteristics and dummy variables for random assignment blocks. This anal- 
ysis also only included the 47 public school sites in the larger Making Pre-K Count study. See Ap- 
pendix B for further details about the analysis. 


Attrition 


The What Works Clearinghouse (WWC) standards, which provide boundaries for acceptable levels 
of attrition for minimizing bias in randomized controlled trials, guided the calculations for overall 
and differential attrition by third grade.’ The WWC provides specific guidelines for judging whether 
the combination of overall and differential individual-level attrition is high and in need ofa baseline 
equivalence testing under an “optimistic” standard for early childhood education studies, reflecting 
the WWC’s assumption that mostattrition in studies ofinterventions results from exogenous factors. 
Under optimistic assumptions, overall attrition up to 40 percent is acceptable when paired with dif- 
ferential attrition levels below 6 percent. Under cautious assumptions, overall attrition up to 40 per- 
centis acceptable when paired with differential attrition levels below 2.6 percent. 


Specifically, of the 2,702 eligible students in the Making Pre-K Count sample (for the confirmatory 
sample and outcome), 1,844 students remained in the New York City Department of Education data 
system, indicating an overall attrition rate of 32 percent. Differential attrition was 1.5 percent, with 
the program group having an attrition rate of 33 percentand the control group havinga rate of 31 
percent.* The High 5s math sample had an overallattrition rate of 30 percentand differential attrition 
rate of 2.0 percent, with the program group having an attrition rate of 29 percent and the control 
group having a rate of 31 percent. Overall and differential attrition rates in Making Pre-K Count and 
Highss fall belowboth optimistic and cautious thresholds for differential attrition (1.5 to 2.0 percent) 
and overall attrition (30 to 32 percent). 


MEASURES 


The research team obtained data on children’s third-grade outcomes from the New York City De- 
partment of Education administrative records, via the Research Alliance for New York City Schools. 
Outcomes included retention in a grade, placement in special education, chronic absenteeism, and 
test scores. Demographic data used as covariates included students’ age, gender, race, and primary 
language at home. 


Estimating impacts on all available outcomes indiscriminately could lead to finding that some esti- 
mates were statistically significant due to chance alone. The team used a multi-tiered approach to 
reduce this likelihood, while preserving the power to identify “true” program impacts. Currently, 


8What Works Clearinghouse (2020). 

‘Across the five outcomes for the Making Pre-K Count full implementation year sample, total attrition ranged from 16 per- 
cent to 34 percent and differential attrition ranged from 0 percent to 2 percent. What Works Clearinghouse labels these as 
low attrition rates. 


there is little consensus in the field of statistics or evaluation on the most appropriate methods for 
adjusting statistical tests to account for multiple comparisons. Moreover, while these statistical ad- 
justments may make it less likely to find false positives, it is not clear that it is worth the tradeoff in 
making it harder to identify true positives. Therefore, rather than correcting for multiple compari- 
sons, the team (1) carefully limited the number of outcomes in its analysis and (2) grouped research 
questions into confirmatory and exploratory categories.° 


The confirmatory outcome for these studies was children’s third-grade math scores, since math skills 
are the direct target of the Making Pre-K Count and High 5s programs. The studies used the New 
York State third-grade standardized math test score as a measure of math skills.° The confirmatory 
questions for these studies were (1) to what extent does Making Pre-K Count affect children’s third- 
grade math test scores; (2) to what extent does High 5s affect children’s third-grade math test scores; 
and (3) to what extent do two years of math enrichment (Making Pre-K Count in pre-K and High 5s 
in kindergarten) affect children’s third-grade math test scores. 


Exploratory research questions focused on (1) outcomes other than math skills and (2) subgroup 
analyses. Other outcomes included those the research team theorized could be indirectly affected by 
the Building Blocks program based on educational and developmental theory (and some empirical 
evidence, albeit more limited than for confirmatory outcome). For example, there is a growing con- 
sensus that math can help build language skills as well as math understanding. Math concepts such 
as countingand shapes expand and enrich vocabulary, as children use language to express and justify 
mathematical thinking.’ In one study, children in Building Blocks classrooms significantly outper- 
formed children in control classrooms on a measure of oral language.* While these outcomes are not 
the explicit focus of the Making Pre-K Count and High 5s programs, the studies were designed not 
only to test the effects of the programs on math skills, but also to test whether early math enrichment 
interventions could affect outcomes in other domains. 


The New York State third-grade standardized English language arts test score was used as a measure 
of reading skills. The research team assessed children’s retention in a grade by whether the student 
was below the expected grade level four years after pre-K.’ The team measured chronic absenteeism 
by whether a student was present 90 percent of the days or less (absent 10 percent or more of school 
days) in third grade.’ Finally, the team measured placement in special education by whether the 


5The research team preregistered confirmatory and exploratory measures as part of the Making Pre-K Count, High 5s, and 
two years of math analytic plans. The preregistered plans can be found at: https://osf.io/obm6va, https://osf.io/ujxnr, and 
https://osf.io/68yxg. 

®These scores were normed to have a mean of 0 and standard deviation of 1 using all New York City test scores for a 
given test (that is, math or English language arts) during a given school year. Children in the soft start year were expected 
to be in third grade in the 2017-2018 academic year and children in the full implementation year were expected to be in 
third grade in the 2018-2019 academic year, so scores were for children taking the test in their expected third-grade year. 
7Ginsburg, Lee, and Boyd (2008). 

®Sarama, Lange, Clements, and Wolfe (2012). 

°Four years after Making Pre-K Count refers to the academic year when students were expected to be in third grade. It refers 
to the 2017-2018 academic year for the soft start year and the 2018-2019 academic year for the full implementation year. 

In a 178-day academic year, a student would be considered chronically absent for missing 18 days or more. If the child 
was previously retained in a grade, chronic absenteeism was collected in whatever grade they were attending four years 
after participating in Making Pre-K Count—the 2017-2018 academic year for the soft start year and the 2018-2019 aca- 
demic year for the full implementation year. 
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student had an Independent Education Program documented by the New York City Department of 
Education four years after participating in Making Pre-K Count. 


The research team classified all subgroup analyses as exploratory because, at the start of Making Pre- 
K Count, there was little available evidence that the impact of Building Blocks varied by baseline 
characteristics of children, teachers, or sites. Subgroup analyses examined whether impacts varied by 
site (community-based organization versus public school) and child characteristics (race or ethnicity, 


language at home, gender, entering skill level). 


Third-Grade Impacts 


d be chapter presents the effects of the Making Pre-K Count prekindergarten (pre-K) math cur- 
riculumand High 5s kindergarten math clubs on children’s third-grade math and literacy test 
scores, retention in a grade, chronic absenteeism, and placement in special education. 


¢ Oneyear of math enrichmentin Pre-K: Though notstatistically significant, Making Pre-K Count 
had a small, positive, longer-term impact on children’s third-grade math test scores (ES = 0.10), 
compared with the pre-K as usual at the control group sites. 


e An additional year of math enrichment in kindergarten: The impact of High 5s on children’s 
third-grade math test scores in public schools, over and above the effect of Making Pre-K Count 
alone, was close to zero and not statistically significant (ES = 0.02). 


e Two years of math enrichment (in pre-K and kindergarten): Making Pre-K Count and High 5s 
together had moderate, statistically significant impacts on children’s math test scores, compared 
with pre-K and kindergarten as usual in public schools (ES = 0.34). 


The finding that two years of enrichment (Making Pre-K Count plus High 5s) had moderate effects 
seems counter-intuitive given the small effects of each of the two interventions separately. This pat- 
tern of results is likely due to differences among the samples of children used in each analysis. Ex- 
ploratory subgroup analyses suggest that early math enrichment may have been particularly benefi- 
cial for children with the mostroom to grow—that is, those children entering pre-K with the weakest 
skills or the lowest test scores. 


The research teamalso estimated the impact of these two early math enrichmentinterventions on other, 
exploratory outcomes. These exploratory analyses suggest that Making Pre-K Count alone and Making 
Pre-K Count plus High 5s reduced chronic absenteeism and improved children’s literacy test scores, 
though findings were not always statistically significant for literacy test scores. The early math inter- 
ventions had an effect close to zero on children’s retention in a grade or placement in special education. 


IMPACTS OF MAKING PRE-K COUNT 


This section presents findings comparing third-grade outcomes of children who received one year of 
math enrichmentin pre-K (Making Pre-K Count) with outcomes of children who received pre-K as 


usual. Table 3.1 presents the pre-K program effects for both the confirmatory outcome—math skills— 
and the exploratory outcomes—literacy test scores, chronic absenteeism, retention in a grade, and 
placement in special education. Appendix C presents the program’s effects on further exploratory 
samples. 


Table 3.1 


Impacts of Making Pre-K Count on Third-Grade Outcomes 


Program Control Difference Standard Effect 
Outcome Measure Group Mean Group Mean (Impact) P-Value Error Size* 
Math? -0.02 -0.12 0.10 0.19 0.08 0.10 
Literacy® 0.02 -0.09 0.11 0.12 0.07 0.11 
Chronic absenteeism (%)° 23.6 32.6 -9.0 0.00 *** 3.0 -0.19 
Retention (%)° 12.5 12.1 0.4 0.84 2.0 0.01 
Special education (%)' 18.6 20.1 -1.5 0.40 1.7 -0.04 
Sample size 
Blocks 16 16 
Sites 35 34 
Students? 945 899 


SOURCE: MDRC calculations based on administrative records from the New York City Department of 
Education, via the Research Alliance for New York City Schools. 


NOTES: Calculations are made using students from the full implementation year sample. 

Bolded outcome is confirmatory, all others are exploratory. 

Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 

The program group received Making Pre-K Count in pre-K. The control group did not receive math 
enrichment and participated in pre-K as usual. 

Impacts were estimated by comparing third-grade outcomes for the group assigned to Making Pre-K Count 
in pre-K with corresponding outcomes for the pre-K-as-usual control group, with an adjustment for selected 
background characteristics and dummy variables for the random assignment blocks. 

Rounding may cause slight discrepencies in sums and differences. 

aEffect size is calculated by dividing the impact of the program (the difference between the means for the 
program group and the control group) by the standard deviation for the control group. 

bCitywide standardized z-score for state third-grade math test. 

¢Citywide standardized z-score for state third-grade English language arts test. 

‘The outcome is defined as whether the student was chronically absent (attended <90 percent of school 
days) in third grade. 

®The outcome is defined as whether the student was below grade level in third grade. It excludes students 
who do not have a valid grade due to enrollment in self-contained special education classrooms. 

‘The outcome is defined as whether the student had an Individualized Education Program (IEP) in third 
grade. 

9The sample size refers to the number of students from the full implementation year sample for which test 
score data were available for math, the study's confirmatory outcome. The analytic sample refers to students 
with any outcome data. For the full implementation year analytic sample, 81 percent have data for math and 
at least 79 percent have data for all other outcomes in the table. 
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e Making Pre-K Count had a positive but not statistically significant impact on children’s third- 
grade math skills, the study’s confirmatory outcome. 


At the end of third grade, the program impacts on math test scores were positive but not statisti- 
cally significant (ES = 0.10, p = 0.19).' Students in control group pre-K sites scored 0.12 standard 
deviations below the citywide average, and students in Making Pre-K Count sites scored 0.02 
standard deviations below the average, a difference of 0.10 standard deviations. This effect is com- 
parable to those found from other curricula implemented at scale and translate to approximately 
12 percent of the achievement gap in fourth grade between low-income children and their high- 
income peers.” 


e The effects of Making Pre-K Count on all exploratory outcomes were in a favorable direction. 
Making Pre-K Count had a positive and marginally statistically significant impact on chil- 
dren’s third-grade literacy skills. The program had a favorable and statistically significant im- 
pact on children’s chronic absenteeism in third grade. The effect on children’s retention in a 
grade or placement in special education was close to zero. 


As with math, at the end of third grade, the program impacts on literacy test scores were positive and 
not statistically significant (ES = 0.11, p = 0.12). Students in control group pre-K sites scored 0.09 
standard deviations below the citywide average in reading, while students in Making Pre-K Count 
sites scored 0.02 standard deviations above the citywide average. This effect on literacy test scores is 
similar in magnitude to the effect on math test scores.’ A child’s reading skills in the third grade have 
long been an important indicator of whether the child completes high school and attends college, 
and numerous policy initiatives around the country focus on improving third-graders’ reading skills 
as a crucial policymaking lever for improving a child’s academic trajectory and later outcomes.‘ The 
effect is comparable to those found from other curricula implemented at scale and translates to ap- 
proximately 15 percent of the achievement gap in third grade between low-income children and their 
high-income peers.” 


'The effect size is calculated by dividing the estimated effect of the program (the difference between the means of the pro- 
gram group and the control group) by the standard deviation for the control group. An effect size of 0.10 here represents 
an improvement in math test scores equal to one-tenth of the standard deviation. 

*Long-term effects from other interventions implemented at scale range from effects of 0.28 on third-grade literacy test 
scores from a social-emotional learning intervention (McCormick et al., 2021) to effects of 0.04 on fourth-grade math skills 
and 0.26 on fifth-grade math skills in a study of Building Blocks (Watts, Duncan, Clements, and Sarama, 2018). Effect sizes 
in the Making Pre-K Count study are standardized measures of the difference in outcomes at the end of third grade for the 
control and program groups. To contextualize these impacts, the research team compared the effect sizes with other avail- 
able standardized data on the difference in the achievement gap between children who are eligible for free or reduced 
price lunch and those who are not eligible. As described in Hill, Bloom, Black, and Lipsey (2008), using National Assess- 
ment of Educational Progress (NAEP) data from 2000 for children at the end of fourth grade, the achievement gap between 
low-income children and their high-income peers was equivalent to 0.85 standardized units for math. The effect of Making 
Pre-K Count on third-grade math scores (0.10) is equivalent to 12 percent of that difference. 

An evaluation of a social-emotional early childhood curriculum in New York City also found statistically significant effects 
on reading but not math test scores, despite the effects on reading and math being similar in magnitude. This suggests 
that the reading test used in New York City may be somewhat more sensitive to program impacts (McCormick et al., 
2021). 

4Snow, Burns, and Griffin (1998); Duncan and Magnuson (2011); Hernandez (2011); Rose and Schimke (2012). 

5Effect sizes in this study are standardized measures of the difference in outcomes at the end of third grade for the control 
and program groups. To calculate the proportion of the achievement gap, the research team compared the effect sizes 
with standardized measures of the difference in outcomes at the end of fourth grade of children who are eligible for free or 
reduced price lunch and those who are not eligible. As described in Hill, Bloom, Black, and Lipsey (2008), using NAEP data 
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Chronic absenteeism is a nationwide problem. In the 2015-2016 academic year, the U.S. Department 
of Education estimated that roughly 16 percent of students nationwide were chronically absent, with 
rates of chronic absenteeism often considerably higher in cities.° In the Making Pre-K Count study, 
approximately 28 percent of third-graders were chronically absent. Among older students, absentee- 
ism is a strong predictor of both high course failure rates and low graduation rates.’ A 2007 study of 
graduation patterns in Chicago Public Schools found absenteeism was eight times more predictive 
of course failure than test scores. * For younger students, research has shown that chronicabsenteeism 
is associated with lower achievement in reading and math and poor socioemotional outcomes, even 
after controlling for a wide range of background characteristics.’ 


Making Pre-K Count had a favorable, statistically significant impact of 9 percentage points on chil- 
dren’s chronic absenteeism in third grade (ES = -o0.19, p < 0.01). Thirty-three percent of students 
who received pre-K as usual were chronically absent in third grade compared with 24 percent of 
students who received math enrichmentin pre-K, which translates to a 28 percent reduction in 
chronic absenteeism." 


Rigorous studies of interventions designed to reduce absenteeism are rare, and those that do exist 
measure outcomes ina variety of different ways. However, a recent randomized control trial of the 
Early Warning Intervention and Monitoring System (EWIMS) gives a senseofthe general magnitude 
of the effects these interventions can have. The EWIMS program includes highly detailed and struc- 
tured guidance for schools, along with a tool to help monitor student attendance and academic per- 
formance; the evaluation indicated that the program reduced chronic absenteeism rates from 14 to 
10 percent (a 28 percent decrease) after one year.'' This effect is comparable to the decrease due to 
Making Pre-K Count. 


Making Pre-K Count did not have an effect, positive or negative, on children’s retention in a grade 
or placement in special education. These outcomes are not specific targets of Making Pre-K Count; 
however, some pre-K programs have found emerging effects on retention in a grade and placement 
in special education in kindergarten, although those effects did not always persist.” 


The research team considers the subgroup analyses to be exploratory, and the Making Pre-K Count 
study was not primarily designed or powered to detect differences in impacts between groups of chil- 
dren or sites. Table 3.2 presents Making Pre-K Count’s impact on the confirmatory outcome (math 
skills) for Hispanic and non-Hispanic students, boys and girls, and students whose primary language 
at home is English and those whose primary language at home is another language. The team tested 
subgroup effects in these groups using the pooled sample of students—students in both the “soft 


from 2000, the income achievement gap was equivalent to 0.74 standardized units for literacy. The effect of Making Pre-K 
Count on third-grade reading scores (0.11) is equivalent to 15 percent of that gap. 

6U.S. Department of Education (2019); Civil Rights Data Collection (2016). 

7Allensworth and Easton (2007); Baltimore Education Research Consortium (2011). 

®Allensworth and Easton (2007). 

®Romero and Lee (2007); Gottfried (2014). 

The effect of Making Pre-K Count on chronic absenteeism began early and continued as children moved through ele- 
mentary school. (See Appendix E.) 

"Faria et al. (2017) 

"Morris et al. (2014); Lipsey et al. (2013). 


Table 3.2 
Impacts of Making Pre-K Count on Third-Grade Math, by Demographics 


Difference 
Program Control Difference Standard Between 
Subgroup Group Mean Group Mean (Impact) P-Value Error Subgroups P-Value 
Race/ethnicity 
Hispanic -0.04 -0.18 0.14 0.02 0.06 0.05 0.65 
Non-Hispanic 0.06 -0.03 0.09 0.31 0.09 
Gender 
Male 0.02 -0.11 0.12 0.09 0.07 0.05 062 
Female -0.03 -0.11 0.07 0.30 0.07 
Home language? 
Non-English -0.03 -0.16 0.13 0.04 0.06 0.05 0.62 
English -0.01 -0.09 0.08 0.25 0.07 
Sample size 
Blocks 16 16 
Sites 35 34 
Students 1,952 1,894 


SOURCE: MDRC calculations based on administrative records from the New York City Department of Education, 
via the Research Alliance for New York City Schools. 


NOTES: Calculations are made using students from the pooled sample (students from both the soft start year 
sample and full implementation year sample). 

Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 

The program group received Making Pre-K Count in pre-K. The control group did not receive math enrichment 
and participated in pre-K as usual. 

Impacts were estimated by comparing third-grade outcomes for the group assigned to Making Pre-K Count in 
pre-K with corresponding outcomes for the pre-K-as-usual control group, with an adjustment for selected 
background characteristics and dummy variables for the random assignment blocks. 

Rounding may cause slight discrepencies in sums and differences. 

4Effect size is calculated by dividing the impact of the program (the difference between the means for the 
program group and the control group) by the standard deviation for the control group. 

>This represents the primary language spoken in the child's home. 


start” year and the full implementation year—in order to maximize power. Table 3.3 shows Making 
Pre-K Count’s impacts on the confirmatory outcome for children entering pre-K with higher and 
lower relative skills. Data on children’s baseline skills were only available for a subset of children in 
the second, fullimplementation year because only a random subsample of children were selected for 
baseline testing. Appendix D presents Making Pre-K Count’s impacts on exploratory outcomes by 
subgroup. 
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Table 3.3 
Impacts of Making Pre-K Count on Third-Grade Math, by Entering Skill Level 


Difference 
Program Control Difference Standard Effect Between 


Subgroup Group Mean Group Mean (Impact) _P-Value Error Size*] Subgroups P-Value 


Entering language skill level” 


High 0.21 0.08 0.14 0.27 0.12 0.14 0.14 0.48 
Low -0.27 -0.55 0.28 0.08 * 0.16 0.28 
Entering self-regulation skill level® 
High 0.08 -0.06 0.14 0.28 0.13 0.14 0.22 0.24 
Low -0.04 -0.40 0.37 0.01 *** 0.13 0.36 
Sample size 
Blocks 16 16 
Sites® 35 34 
Students® 298 295 


SOURCE: MDRC calculations based on administrative records from the New York City Department of Education, via the Research Alliance for New 
York City Schools. 


NOTES: Calculations are made using students from the full implementation year sample. 

Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 

The program group received Making Pre-K Count in pre-K. The control group did not receive math enrichment and participated in pre-K as usual. 

Impacts were estimated by comparing third-grade outcomes for the group assigned to Making Pre-K Count in pre-K with corresponding outcomes 

for the pre-K-as-usual control group, with an adjustment for selected background characteristics and dummy variables for the random assignment 
blocks. 

Rounding may cause slight discrepencies in sums and differences. 

aEffect size is calculated by dividing the impact of the program (the difference between the means for the program group and the control group) by 
the standard deviation for the control group. 

>Children's language skills were measured using the Receptive One-Word Picture Vocabulary Test (ROWPVT-4; Martin and Brownell, 2011), 
administered at pre-K entry in the fall of 2014. 

°Children's self-regulation skills were measured using the Preschool Self-Regulation Assessment (PSRA; Smith-Donald, Raver, Hayes, and 
Richardson, 2007), administered at pre-K entry in the fall of 2014. 

dAs few as 33 program and control sites are represented for certain subgroups. 

®There are 296 students in the program group and 294 students in the control group for the language subgroup. 


e The effects of Making Pre-K Count were similar in magnitude across different subgroups of 
children. The effects were similar across Hispanic and non-Hispanic students, across boys and 
girls, and across students whose primary language at home was English and those whose pri- 
marty language at home was not English. 


A prior study found that the Building Blocks curriculum had larger impacts for Black students; how- 
ever, those studies includeda limited sample of Hispanic students (22 percent of study participants).’’ 
Making Pre-K Countincludeda larger sample of Hispanic children (54 percent ofstudy participants), 
with the rest of the sample identifying primarily as non-Hispanic Black (37 percent). By third grade, 
Making Pre-K Count had positive and statistically significant effects on math skills (ES = 0.14, p = 
0.02) for Hispanic students. The effect on math test scores is equivalent to 16 percent of the achieve- 
ment gap between low-income children and their high-income peers. The impact on non-Hispanic 
children’s math scores (ES = 0.09) was ofa similar magnitude, although not statistically significant." 
Making Pre-K Count had a positive but not statistically significant effect on math skills (ES = 0.10, 
p = 0.42) for Black boys. 


Making Pre-K Count had a positive and statistically significant effecton boys’ math scores (ES = 0.12, 
p = 0.09). The effect on girls’ math scores was similar in magnitude but not statistically significant 
(ES = 0.08, p = 0.30). Making Pre-K Count had a positive and statistically significant effect on stu- 
dents whose primary language at home was not English (ES = 0.14, p = 0.04) and a positive but not 
statistically significant effect on students whose primary language at home was English (ES = 0.08, 


p = 0.25). 


e Making Pre-K Count had statistically significant impacts on third-grade math scores for chil- 
dren entering the study with lower skills, with large and positive effects for children entering 
pre-K with lower self-regulation or language ability. 


The research team assessed children’s baseline pre-K skills for only a subset of children, and only in 
the second year of implementation. These analyses include only those children who were randomly 
selected for this baseline assessment. 


Making Pre-K Count had large and statistically significantimpacts on math test scores for those chil- 
dren rated as having lower self-regulation skills (being more impulsive) or weaker language skills 
when entering pre-K in the fall (ES = 0.36 and 0.28, respectively). The program had small, positive, 
and not statistically significant impacts for children with stronger skills in the fall of pre-K (ES = 0.14 
and 0.14). 


*8Clements et al. (2011). 

‘Alternative ways of testing the effect of Making Pre-K Count for children in different racial or ethnic subgroups 
showed substantively similar results. Tests comparing random assignment blocks that served a majority of students 
who were Hispanic with random assignment blocks that served a majority of non-Hispanic students showed the same 
pattern of effects on math skills (ES = 0.15, p = 0.10 for Hispanic blocks and ES = 0.03, p = 0.74 for non-Hispanic 
blocks). 


e Effects of Making Pre-K Count on test scores were generally larger in magnitude for children 
in public schools. 


Making Pre-K Count was implemented in both public school and community-based sites. Findings 
from pre-K and kindergarten suggest that Making Pre-K Count had positive effects on math out- 
comes in public school sites and executive functioning outcomes in community-based sites. Table 
3.4 presents Making Pre-K Count’s impacts on third-grade outcomes for children who attended pre- 
K in public schools and for those who attended pre-K in community-based sites. 


For children who attended pre-K in public schools, Making Pre-K Count had positive marginally 
significant effects on third-grade math scores (ES = 0.16, p = 0.10) and statistically significant effects 
on third-grade reading scores (ES = 0.18, p = 0.03) and chronic absenteeism (ES = -0.18, p = 0.02). 
These test score effects are equivalent to 19 and 24 percent of the achievement gap between low- 
income children and their high-income peers. 


For children who attended pre-K in community-based sites, Making Pre-K Count had small, nega- 
tive, and not statistically significant effects on third-grade math and literacy scores. The program had 
favorable and statistically significant effects on third-grade chronic absenteeism (ES = -0.23, p = 0.06) 
and retention in a grade (ES = -0.15, p = 0.06) for children who attended pre-K in community-based 
sites. The program hada statistically significantly larger effect on retention in community-based sites 
than it did on the same outcome in public schools. 


The High 5s kindergarten math club was implemented only in public school sites. Therefore, the 
effects of the High 5s program described below are on top of Making Pre-K Count’s positive impacts 
on outcomes for children in public schools. Importantly, control group children who attended pre- 
K in public schools had lower third-grade math test scores (-0.20) than control group children who 
attended pre-K in community-based organizations (0.06), suggesting that the public school children 
may have also been a higher-risk group. 


IMPACTS OF HIGH 5s 


High 5s operated in the 24 public schools that had implemented the Making Pre-K Count program 
in pre-K. All children in the High 5s study stayed in the same school for pre-K and kindergarten and 
therefore had atleast one year of math enrichment (Making Pre-K Count). The research team ran- 
domlyassigned halfofthe childrenwho received Making Pre-K Countto an additional, aligned math 
enrichment intervention (High 5s). 


This section presents findings comparing third-grade outcomes for children assigned to two years of 
math enrichment (Making Pre-K Count plus High 5s) with third-grade outcomes for children as- 
signed to one year of math enrichment (Making Pre-K Count only). The effects of High 5s are in 
addition to Making Pre-K Count’s positive impacts on outcomes of children in public schools. 


The research team also estimated impacts in public schools and community-based sites across implementation years 
using the pooled sample. The results showed similar effects to those in the full implementation year. 
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Outcome Measure 


Math? 


Literacy® 


Chronic absenteeism (%)" 


Retention (%)° 


Special education (%)' 


Sample size 
Blocks 
Sites 
Students? 


Table 3.4 


Impacts of Making Pre-K Count on Third-Grade Outcomes, 
by Venue (Community-Based Organization Versus Public School) 


Community Based-Organization Public School 


Control Difference Control Difference Between 


Group Mean P-Value Group Mean Subgroups 


Difference 
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P-Value 
0.28 
0.17 
0.86 
0.03 tt 


0.59 


(continued) 


Table 3.4 (continued) 


SOURCE: MDRC calculations based on administrative records from the New York City Department of Education, via the Research Alliance for New York 
City Schools. 


NOTES: Calculations are made using students from the full implementation year sample. 

Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. Statistically significant differences in impact 
estimates across different subgroups are indicated as follows: ttt = 1 percent; tt = 5 percent; t = 10 percent. 

The program group received Making Pre-K Count in pre-K. The control group did not receive math enrichment and participated in pre-K as usual. 

Impacts were estimated by comparing third-grade outcomes for the group assigned to Making Pre-K Count in pre-K with corresponding outcomes for 
the pre-K-as-usual control group, with an adjustment for selected background characteristics and dummy variables for the random assignment blocks. 

Rounding may cause slight discrepencies in sums and differences. 

aEffect size is calculated by dividing the impact of the program (the difference between the means for the program group and the control group) by the 
standard deviation for the control group. 

bCitywide standardized z-score for state third-grade math test. 

¢Citywide standardized z-score for state third-grade English language arts test. 

dThe outcome is defined as whether the student was chronically absent (attended <90 perecent of school days) in third grade. 

€The outcome is defined as whether the student was below grade level in third grade. It excludes students who do not have a valid grade due to 
enrollment in self-contained special education classrooms. 

‘The outcome is defined as whether the student had an Individualized Education Program (IEP) in third grade. 

9The sample size refers to the number of students from the full implementation year sample for which test score data were available for math, the 
study's confirmatory outcome. The analytic sample refers to students with any outcome data. For the full implementation year analytic sample, 81 percent 
have data for math and at least 79 percent have data for all other outcomes in the table. 
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e The impact of High 5s, which was implemented in public schools only, on children’s third- 
grade outcomes, over and above the effect of Making Pre-K Count alone, was close to zero and 


not statistically significant. (See Table 3.5.) 


While High 5s had short-term effects on children’s math skills at the end of kindergarten, its effects 
on children’s third-grade math and literacy scores, absenteeism, retention in a grade, and placement 
in special education, above and beyond the effects of Making Pre-K Count, were close to zero and 


not statistically significant. 


Table 3.5 


Impacts of High 5s on Third-Grade Outcomes 


Program Control Difference Standard Effect 
Outcome Measure Group Mean Group Mean (Impact) P-Value Error Size* 
Math? -0.04 -0.06 0.02 0.82 0.08 0.02 
Literacy® -0.03 0.02 -0.05 0.54 0.08 -0.05 
Chronic absenteeism (%)° 24.3 28.9 -4.5 0.23 3.8 -0.10 
Retention (%)° 13.8 12.4 1.4 0.62 2.8 0.04 
Special education (%)' 14.6 14.9 -0.4 0.90 3.0 -0.01 
Sample size 
Sites 24 24 
Students? 226 230 


SOURCE: MDRC calculations based on administrative records from the New York City Department of 
Education, via the Research Alliance for New York City Schools. 


NOTES: Bolded outcome is confirmatory, all others are exploratory. 

Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 

The program group received Making Pre-K Count in pre-K and the High 5s in kindergarten. The control 
group recieved Making Pre-K Count in pre-K but kindergarten as usual. 

Impacts were estimated by comparing third-grade outcomes for the group assigned to Making Pre-K 
Count in pre-K and High 5s in kindergarten with corresponding outcomes for the control group that did not 
recieve Making Pre-K Count and had kindergarten as usual, with an adjustment for selected background 
characteristics and dummy variables for the random assignment blocks. 

Rounding may cause slight discrepencies in sums and differences. 

4Effect size is calculated by dividing the impact of the program (the difference between the means for 
the program group and the control group) by the standard deviation for the control group. 

Citywide standardized z-score for state third-grade math test. 

cCitywide standardized z-score for state third-grade English language arts test. 

4The outcome is defined as whether the student was chronically absent (attended <90 percent of 
school days) in third grade. 

®The outcome is defined as whether the student was below grade level in third grade. It excludes 
students who do not have a valid grade due to enrollment in self-contained special education classrooms. 

fThe outcome is defined as whether the student had an Individualized Education Program (IEP) in third 
grade. 

9The sample size refers to the number of students from the High 5s sample for which test score data 
were available for math, the study's confirmatory outcome. The analytic sample refers to students with any 
outcome data. For the High 5s analytic sample, 82 percent have data for math and at least 82 percent 
have data for all other outcomes in the table. 
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IMPACT OF TWO YEARS OF EARLY MATH ENRICHMENT 


In addition to examining the impacts of each program separately, the Making Pre-K Count and High 
5s studies were designed to allow for a test of the effect of two years of math enrichment in pre-K and 
kindergarten, compared with pre-K and kindergarten as usual. This section presents findings com- 
paring third-grade outcomes for children who received two years of math enrichment (the Making 
Pre-K Count plus High 5s program group) with third-grade outcomes for children who received no 
math enrichment (the pre-K-and-kindergarten-as-usual control group). Because this comparison 
builds off the High 5s program, it includes only children who were eligible for High 5s—that is, chil- 
dren who attended pre-K and kindergarten in the same public school. 


e Two years of enriched math instruction in public schools led to positive and statistically sig- 
nificant effects on children’s third-grade math scores (ES = 0.34), the confirmatory outcome 
for the study, when compared with pre-K and kindergarten as usual. 


The research team observed only small impacts of Making Pre-K Count on children’s third-grade 
outcomes in public schools (for example, ES = 0.16 for math skills in public schools) and found that 
High 5s had no added benefit. Nevertheless, the effect of two years of early math enrichment on out- 
comes of students who attended the same public school for pre-K and kindergarten was substantially 
larger than those estimates combined, when compared with a comparable sample of public school 
students who received no math enrichment in pre-K or kindergarten. (See Table 3.6.) 


Table 3.6 
Impacts of Making Pre-K Count and High 5s on Third-Grade Outcomes 


Program Control Difference Standard Effect 
Outcome Measure Group Mean Group Mean ___(Impact) P-Value Error Size* 
Math? -0.08 -0.42 0.34 0.01 ** 0.14 0.34 
Literacy® -0.07 -0.34 0.27 0.02 ** 0.11 0.29 
Chronic absenteeism (%)° 23.4 32.8 -9.5 0.04 ** 45 -0.20 
Retention (%)° 14.0 11.0 3.0 0.36 3.3 0.10 
Special education (%)' 14.3 19.6 -5.3 0.22 4.3 -0.13 
Sample size 
Blocks 11 10 
Sites 24 22 
Students? 226 255 


(continued) 
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Table 3.6 (continued) 


SOURCE: MDRC calculations based on administrative records from the New York City Department of 
Education, via the Research Alliance for New York City Schools. 


NOTES: Bolded outcome is confirmatory, all others are exploratory. 

Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 

The program group received Making Pre-K Count in pre-K and the High 5s in kindergarten. The control 
group recieved pre-K and kindergarten as usual. 

Impacts were estimated by comparing third-grade outcomes for the group assigned to Making Pre-K 
Count in pre-K and High 5s in kindergarten with corresponding outcomes for the control group that did not 
recieve Making Pre-K Count and had kindergarten as usual, with an adjustment for selected background 
characteristics and dummy variables for the random assignment blocks. 

Rounding may cause slight discrepencies in sums and differences. 

aEffect size is calculated by dividing the impact of the program (the difference between the means for the 
program group and the control group) by the standard deviation for the control group. 

5Citywide standardized z-score for state third-grade math test. 

¢Citywide standardized z-score for state third-grade English language arts test. 

dThe outcome is defined as whether the student was chronically absent (attended <90 percent of school 
days) in third grade. 

©The outcome is defined as whether the student was below grade level in third grade. It excludes students 
who do not have a valid grade due to enrollment in self-contained special education classrooms. 

'The outcome is defined as whether the student had an Individualized Education Program (IEP) in third 
grade. 

9The sample size refers to the number of students from the two years of math sample for which test score 
data were available for math, the study's confirmatory outcome. The analytic sample refers to students with 
any outcome data. For the two years of math analytic sample, 82 percent have data for math and at least 83 
percent have data for all other outcomes in the table. 


e Two years of early math enrichment also led to a positive and statistically significant impact 
on literacy test scores (ES = 0.29) and an impact of approximately 9 percentage points on 
chronic absenteeism (ES = -0.20), both exploratory outcomes. 


The effect of two years of math enrichment on retention ina grade or placement in special education 
were small and not statistically significant. 


The relatively large impacton both math and reading testscores atthe end of third grade for children 
who were offered two years of early math enrichment does not clearly align with Making Pre-K 
Count’s relatively modest impacts and Highs’ lack ofa statistically significant impact on children’s 
third-grade outcomes. 


As described earlier, the impact of Making Pre-K Count appeared to be largest for students entering 
pre-K with weaker skills—possibly because they had the most room to grow. The research team ex- 
amined the sample of children used in the analysis of two years of early math enrichment (students 
who stayed in the same public school for pre-K and kindergarten) and found that it may have in- 
cluded more low-performing children than the full study sample. 


Table 3.7 presents the average third-grade standardized math test score (a score of zero represents 


the city average) of control group children in six differentsamples: (1) the full Making Pre-K Count 
study sample, (2) the sample of children who attended pre-K in community-based sites, (3) the 
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Table 3.7 


Control Group Average Third-Grade Math Scores, by Sample 


Sample (Number of Students) Mean 


a 


Full Making Pre-K Count sample control group (899) -0.12 
Full Making Pre-K Count community-based organization sample control group (267) 0.06 
Full Making Pre-K Count public school sample control group (632) -0.20 
Making Pre-K Count plus High 5s sample control group (255) -0.42 
Low entering language skills sample control group>(154) -0.55 
Low entering self-regulation skills sample control group* (136) -0.40 


SOURCE: MDRC calculations based on administrative records from the New York City Department of 
Education, via the Research Alliance for New York City Schools. 


NOTES: The sample sizes refer to the analytic sample for the study's confirmatory outcome, math. 

aCitywide standardized z-score for state third-grade math test. 

bChildren's language skills were measured using the Receptive One-Word Picture Vocabulary Test 
(ROWPVT-4; Martin and Brownell, 2011), administered at pre-K entry in the fall of 2014. 

¢Children's self-regulation skills were measured using the Preschool Self-Regulation Assessment (PSRA; 
Smith-Donald, Raver, Hayes, and Richardson, 2007), administered at pre-K entry in the fall of 2014. 


sample of children who attended pre-K in public schools, (4) the sample of children in the analysis 
of two years of early math enrichment, (5) the sample of children with low language skills, and (6) 
the sample of children with low self-regulation skills. Children in the control group represent how 
children in the program group would have behaved or performed had they not received Making 
Pre-K Count, High 5s, or both. 


Table 3.7 further shows that control group children in the full study sample performed 0.12 standard 
deviations below the citywide average in math at the end of third grade. The students who had at- 
tended pre-K in community-based sites scored 0.06 standard deviations above the citywide average 
in third-grade math, and students who had attended pre-K in public schools scored -o.20 standard 
deviations below the citywide average in third-grade math—suggesting that children in the public 
school sample were lower performing than those in the community-based site sample. Across New 
York City’s pre-K systems, public schools tend to serve a population of children from higher-income 
families. However, the Making Pre-K Countand Highss pre-Ks study samples may nothave reflected 
this trend because they were drawn specifically from community school districts serving children 
from families with low incomes.'° 


Importantly, however, the control group students who were eligible for the High 5s and thus two 
years of early math enrichment (that is, those who attended pre-K and kindergarten in the same 


public school) scored even lower on standardized math tests than the overall public school control 
group. The control group students in this sample scored -o.42 standard deviations belowthe citywide 


Reid, Melvin, Kagan, and Brooks-Gunn (2019). 
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average in third-grade math, which is similar to the third-grade performance of the children in both 
the low self-regulation skills and low language skills groups. 


In other words, it appears that the children who attended pre-K in public schools and stayed in those 
same schools into kindergarten may have had the lowest test scores and the most room to grow. 
Therefore, the children assigned to the group that received two years of early math enrichment may 
have been poised to benefit most from the interventions. 


It is possible that the skills of the students in the control group declined over time due to factors 
unrelated to the intervention. Nevertheless, the research team explored a number of different hy- 
potheses to explain the larger-than-expected impacts for this group of children (presented in Appen- 
dix F) and did not find evidence for these alternative explanations. The children and elementary 
schools in the program and control groups did not differ from each other at baseline (shown in Ap- 
pendix Tables A.1and F.1). 
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4 


Conclusions 


he Making Pre-K Count and High 5s studies were designed to understand the potential of en- 

hanced early math instruction to produce long-term impacts. To date, few studies have tried to 
rigorously assess the impact of improving early math skills on later outcomes. Findings from the few 
prior studies ofthe long-term effects of enhanced early math instruction have been mixed, with some 
finding positive and some null effects in the years after prekindergarten (pre-K).' 


Four years after pre-K, the long-term effect of Making Pre-K Count on math, the main target of the 
program and the study’s confirmatory outcome, was small and positive but not statistically signifi- 
cant in the main confirmatory sample, suggesting that the program was as effective in the long term 
as existing practice in pre-K classrooms in New York City in 2015, and potentially slightly better. This 
was true even though all students in the study attended pre-K and, at the time the program was im- 
plemented, there was a growing emphasis on early math instruction in New York City schools. Even 
children in the control group received more math instruction than had been typical in previous stud- 
ies of early math interventions. 


The effect of High 5s on third-grade outcomes was close to zero. Despite the moderate effects on 
children’s math skills found in the year that the program was implemented, the program’s effect on 
these skills was not sustained three yearslater. High 5s was a supplemental math enrichment program 
delivered in kindergarten outside of instructional time and designed to align with and extend chil- 
dren’s experiences in pre-K. Building off the “sustaining environments” hypothesis, High 5s was in- 
tended to sustain children’s math enrichment experiences in pre-K into kindergarten.* However, it 
was not expected to align closely with day-to-day classroom content in later grades. It is possible that 
this lack of connection to classroom instruction in future years was related to the lack of sustained 
impacts for the program alone. 


High 5s also did not lead to impacts on other, non-math outcomes in kindergarten. It is possible that 
effects in these domains are needed to maintain the early intervention’s impacts, as posited by Heck- 
man, Stixrud, and Urzua.’ Finally, exploratory analyses of Making Pre-K Countsuggestthat the early 
math enrichment interventions are most effective for children with the most room to grow. Because 
High 5s served only children who received Making Pre-K Count in pre-K, all children in the sample 
already had relatively strong math skills. Children in the High 5s control group (those who had 


‘Dumas, McNeish, Sarama, and Clements (2019); Rittle-Johnson, Fyfe, Hofer, and Farran (2016). 
*Bailey, Jenkins, and Alvarez-Vargas (2020). 
SHeckman, Stixrud, and Urzua (2006). 
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received Making Pre-K Count in pre-K but continued with kindergarten as usual) had third-grade 
test scores near to the citywide mean (mean = -0.06). 


The effect of Making Pre-K Count in pre-K plus High 5s in kindergarten on math outcomes, com- 
pared with not receiving any math enrichmentin pre-K or kindergarten, was moderate and statisti- 
cally significant. When children received two years of math enrichment, the effects on test results 
were equivalent to approximately 40 percent of the achievement gap in fourth grade between chil- 
dren from families with low incomes and their peers from families with high incomes.* The finding 
that two years of enrichment (Making Pre-K Count plus High 5s) had moderate effects seems coun- 
ter-intuitive given the small effects of each of the two interventions separately. This pattern of results 
is likely due to differences among the samples of children in each analysis. Exploratory subgroup 
analyses suggest that early math enrichment may have been particularly beneficial for children with 
the most room to grow, and these children were more prevalent in the combined Making Pre-K 
Count and High 5s sample. 


Making Pre-K Count had fairly large impacts on third-grade math and literacy test scores—ranging 
from one-quarter to over a third ofa standard deviation—for children entering pre-K with the lowest 
language and self-regulation scores on standardized assessments and ratings. For example, children 
at sites in the control group (those who received pre-K as usual) who were rated as having higher 
impulsivity at the start of pre-K by evaluators blinded to the child’s treatment status had third-grade 
math test scores 0.40 standard deviations below the citywide mean. The Making Pre-K Count pro- 
gram raised these third-grade test scores up to the citywide average (-0.04) for a comparable group 
of children with similar high impulsivity ratings at the start of pre-K. This pattern of long-term effects 
supports the “academic risk” hypothesis, which posits that early interventions may have the largest 
effects for children with the most room to grow.” 


Making Pre-K Count also had small, positive and not statistically significant effects on literacy test 
scores four years after the pre-K year, suggesting that the program was at least as effective as existing 
practice in pre-K classrooms in New York City in 2015. These findings align with earlier studies of 
the Building Blocks curriculum that found it had impacts on aspects of children’s language ability 
and with Duncan and colleagues’ correlational study showing that early math skills are a strong pre- 
dictor of third-grade reading test scores.° 


Making Pre-K Count alone and the two years of math enrichment also had effects on chronic absen- 
teeism. The earlymath enrichmentreduced chronic absenteeism by aboutg percentage points across 
public schools and community-based organizations. These effects are substantively meaningful: In 
the present studies, Making Pre-K Count reduced rates of chronic absenteeism from approximately 
33 percent of third-graders in the control group to 24 percent in program group. For younger 


‘Effect sizes in this study are standardized measures of the difference in outcomes at the end of third grade for the control 
and program groups. To contextualize these impacts, the research team compared the effect sizes with other available 
standardized data on the difference in achievement between children who are eligible for free or reduced price lunch and 
those who are not eligible. As described in Hill, Bloom, Black, and Lipsey (2008), using National Assessment of Educa- 
tional Progress data from 2000 for children at the end of fourth grade, the achievement gap between those eligible for free 
or reduced price lunch and those not eligible was equivalent to 0.85 standardized units for math. The effect of two years of 
math enrichment on third-grade math scores (0.34) is equivalent to 40 percent of that difference. 

5Hamre and Pianta (2005); Hamre and Pianta (2001). 

®Sarama, Lange, Clements, and Wolfe (2012); Duncan et al. (2007). 
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students, chronic absenteeism is associated with lower achievement in reading and math, and poor 
socioemotional outcomes, even after controlling for a wide range of background characteristics.’ Re- 
ducing absenteeism by 9 percentage points for third-graders citywide in New York City could lead 
to over 7,000 fewer chronically absent third-graders per year.* 


For young children, attendance is generally thought to be a function of the family’s engagement with 
school. While itis unclearhowa teacher-providedinstructional program during the school day could 
change families’ attitudes toward school, earlier findings demonstrated that Making Pre-K Count 
had a positive effect on children’s attitudes toward math in kindergarten. Making Pre-K Count may 
thus have led families to either see school or their children’s excitement aboutschool more positively, 
leading to higher attendance and improved engagement with school. Regardless of the mechanism, 
chronic absenteeism is negatively associated with later academic achievement.’ Students who are 
chronically absent in pre-K and kindergarten have lower test scores by third grade. '° Chronic absen- 
teeism can be considered a red flag for children at risk of performing poorly in school, and Making 
Pre-K Count’s impacts on this high-risk group of children align with the larger pattern of results 
showing the largest effects for children with the greatest needs. 


The sustained effects on chronic absenteeism, in combination with Making Pre-K Count’s earlier 
impacts on children’s executive functioning and attitudes toward math, suggest that early math in- 
terventions can have spillover effects into non-cognitive outcomes that are not usually assessed in 
long-term intervention studies. These spillover effects lend support to the hypothesis posited by 
Heckman, Stixrud, and Urzua that impacts on non-cognitive outcomes may help maintain longer- 
term gains in cognitive domains, although the present analyses do not directly test whether effects 
on non-cognitive outcomes mediate the relationship between the intervention and effects on test 
scores.'! Recent findings from a study examining the long-term effects of the Boston pre-K program 
show that impacts in non-cognitive domains could have potential value for sustaining the effects of 
pre-K programming into adulthood. '’ The researchers found that the program had short-term effects 
on student behavior but not on test scores, and ultimately had long-term impacts on high school 
graduation and college enrollment. 


The findings from the Making Pre-K Count and High 5s studies contribute to growing evidence 
about the importance of early math instruction. They suggest that a well-implemented, evidence- 
based early math enrichment program has the potential to improve academic achievement over the 
longer term as well as outcomes in other domains—not just math skills. The effects across a number 
of domains, including non-cognitive domains, also suggest that early math enrichment could poten- 
tially lead to even longer-term effects on students’ outcomes as they move into middle and high 
school. 


7Romero and Lee (2007); Gottfried (2014). 

®New York City had 78,141 third-graders in 2019-2020 (New York State Education Department, 2020). An analysis by New 
York University estimated that 22.8 percent of students were chronically absent in 2018 (Research Alliance for New York 
City Schools, 2019). According to those numbers, an estimated 17,816 third-graders would be chronically absent. After a 
reduction of chronic absenteeism by 9 percentage points, an estimated 10,784 third-graders would be chronically absent. 
®Romero and Lee (2007); Ansari and Purtell (2018); Ehrlich, Gwynne, and Allensworth (2018); Simon, Nylund-Gibson, Gott- 
fried, and Mireles-Rios (2020). 

10 Connolly and Olson (2012). 

“Heckman, Stixrud, and Urzua (2006). 

"Gray-Lobe, Pathak, and Walters (2021). 
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Appendix A 


Sample Descriptions and 
Baseline Equivalence of Children 
Across Program and Control Groups 


his appendix describes confirmatory and exploratory samples for the Making Pre-K Count and 
High 5s studies. It also lays out the baseline equivalence of children for each sample. More infor- 
mation about the sample selection is available in previous reports.! 


MAKING PRE-K COUNT 


The Making Pre-K Count study tested the effects of an evidence-based pre-K math curriculum 
(Building Blocks), supported by two years of teacher training and in-classroom coaching, that was 
implemented in 69 pre-K sites in New York City.’ In the study, whole pre-K sites were randomly 
assigned to receive the evidence-based math curriculum plus teacher training and coaching (n = 35), 
or to continue pre-K as usual (n = 34). The effects of Making Pre-K Count alone were estimated on 
one confirmatory sample (full implementation year sample) and four exploratory samples (full im- 
plementation year kindergarten analytic subsample, soft start year sample, soft start year consented 
subsample, and pooled sample). Each sample is described in greater detail below. 


Full Implementation Year Sample 


The Making Pre-K Count pre-Ks served children in two distinct years during the time of the study. 
The first was a soft start year to help teachers become familiar with the curriculum and receive train- 
ing. Childrenin the second year of the program’s implementation have been the main focus of the 
Making Pre-K Count study to date and are considered the confirmatory sample for this analysis. 
Administrative records are available for all children in the full implementation year sample still in 
the New York City school system. A total of 2,819 children were on the rosters in the 173 classrooms. 
Of those, 2,702 completed consent forms to participate in the study and were eligible for follow-up 
assessments.’ Of those, 2,277 had third-grade data on any outcome, with each outcome varying in the 
amount of available data. Using the confirmatory outcome, math skills, a Wald test of joint signifi- 
cance indicated that the two groups of children were not systematically different along the available 
baseline demographic characteristics. (See Table A.1.) Wald tests using the samples for the other four 
exploratory outcomes (not shown) yielded the same result: that the two groups of children were not 
systematically different based on demographic characteristics. A test using the full implementation 
year sample (n = 2,702), including those who did not have third-grade data, yielded similar results. 


Full Implementation Year Kindergarten Analytic Subsample 


While the main focus of the study is the full implementation year sample, past analyses have only 
included a subsample of children who were randomly assessed at certain timepoints. To produce 
aligned estimates across time, the research team included an analysis of the same sample used in the 


‘Morris, Mattera, and Maier (2016); Mattera, Jacob, and Morris (2018). 

2Clements and Sarama (2013). 

8Students had to be born before 2010 or at least 4 years old to be eligible to participate. Students in public school pre-K 
programs were assumed to have met this age requirement in order to be enrolled in pre-K; birth dates for students enrolled 
in community-based pre-K programs were used to determine age eligibility for Making Pre-K Count. Of the 2,717 partici- 
pants who consented to participate, 2,702 were age eligible. 
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Appendix Table A.1 


Baseline Equivalence: Making Pre-K Count Versus Pre-K as Usual 


Program Control 

Characteristic Group Group 
Child demographics 
Race and ethnicity (%) 

Hispanic 52.4 55.6 

Non-Hispanic White 6.8 1.5 

Non-Hispanic Black 36.1 37.9 

Other/multiracial® 4.8 5.0 
Female (%) 52.3 52.1 
Home language (%)° 

English 77.1 66.0 
Age* 4.18 4.19 
Parent demographics 
Highest level of education 

At least high school diploma/GED (%) 75.5 71.2 


Sample size® 945 899 


SOURCE: MDRC calculations from administrative records from the New York City Department of 
Education, via the Research Alliance for New York City Schools. 


NOTES: Calculations are made using students from the full implementation year sample. 

The program group received Making Pre-K Count in pre-K. The control group did not receive 
math enrichment and participated in pre-K as usual. 

GED = General Educational Development certificate. 

Rounding may cause slight discrepancies in sums and differences. 

aOther includes Asian, Native Hawaiian/Pacific Islander, and American Indian/Alaska Native. 

>This represents the primary language spoken in the child's home. 

This is the age at the beginning of pre-K as of September 1, 2014. 

dA Wald test was used to determine whether there was a systematic difference between the two 
samples based on the characteristics included in this table. 


®For the parent demographics, n=921 for the program group and n=862 for the control group. 


Making Pre-K Count kindergarten impact report, (students who were assessed at the end of kinder- 
garten).* Of the 1,325 students included in this analysis, 1,180 had third-grade data available on any 
outcome, with each outcome varying in the amount of available data. A sensitivity check was con- 
ducted to replicate the Making Pre-K Count kindergarten impact analysis with the full implementa- 
tion year kindergarten analytic subsample from third grade—that is, those children that were still 
able to be tracked into third grade. Results of the kindergarten impact analysis were similar in mag- 
nitude, direction, and statistical significance using this sample. A Wald test of joint significance using 
the math sample indicated that the two groups of children were not systematically different based on 
demographics. (See Table A.2.) 


4Mattera, Jacob, and Morris (2018). 
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Appendix Table A.2 


Baseline Equivalence: Making Pre-K Count Versus Pre-K as Usual 
for Full Implementation Year Kindergarten Analytic Subsample 


Program Control 

Characteristic Group Group 
Child demographics 
Race and ethnicity (%) 

Hispanic 54.0 56.6 

Non-Hispanic White 6.1 1.8 

Non-Hispanic Black 35.0 36.4 

Other/multiracial* 4.9 5.2 
Female (%) 52.1 53.6 
Home language (%)° 

English 76.2 64.2 
Age* 4.18 4.17 
Parent demographics 
Highest level of education 

At least high school diploma/GED (%) 74.5 68.9 
Joint test of difference between groups (F-value = 0.04) 


Sample size® A474 500 


SOURCE: MDRC calculations from administrative records from the New York City Department of 
Education, via the Research Alliance for New York City Schools. 


NOTES: GED = General Educational Development certificate. 

The program group received Making Pre-K Count in pre-K. The control group did not receive 
math enrichment and participated in pre-K as usual. 

Rounding may cause slight discrepancies in sums and differences. 

aOther includes Asian, Native Hawaiian/Pacific Islander, and American Indian/Alaska Native. 

>This represents the primary language spoken in the child's home. 

°This is the age at the beginning of pre-K as of September 1, 2014. 

dA Wald test was used to determine whether there was a systematic difference between the two 
samples based on the characteristics included in this table. 


€For the parent demographics, n=462 for the program group and n=479 for the control group. 


Soft Start Year Consented Subsample 


Although the students who were in pre-K during the 2013-2014 softstart year were not the main focus 
of this study, an effort was made to collect consent from them. Of the estimated 3,120 students in 
Making Pre-K Count pre-Ks that year, 1,911 (61 percent) consented to participate in the study and 
have their data tracked over the years. By third grade, 1,520 of these students had third-grade data 
available on any outcome, with each outcome varying in the amount of available data. A Wald test 
of joint significance using the math sample indicated that the two groups of children were not sys- 
tematically different based on demographics. (See Table A.3.) 
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Appendix Table A.3 


Baseline Equivalence: Making Pre-K Count Versus Pre-K as Usual 
for Soft Start Year Consented Subsample 


Program Control 
Characteristic Group Group 
Child demographics 
Race and ethnicity (%) 
Hispanic 53.0 52.2 
Non-Hispanic White 6.1 1.6 
Non-Hispanic Black 37.0 42.0 
Other/multiracial* 4.0 42 
Female (%) 50.4 53.3 
Home language (%)° 
English 69.7 65.4 
Age® 4.22 4.20 
‘Joint test of difference between groupss si(‘i‘ééé!t!)})©6(Fvalue H0.03) 0 C~™S 


Sample size 657 638 


SOURCE: MDRC calculations from administrative records from the New York City Department of 
Education, via the Research Alliance for New York City Schools. 


NOTES: The program group received Making Pre-K Count in pre-K. The control group did not 
receive math enrichment and participated in pre-K as usual. 

Rounding may cause slight discrepancies in sums and differences. 

aOther includes Asian, Native Hawaiian/Pacific Islander, and American Indian/Alaska Native. 

>This represents the primary language spoken in the child's home. 

This is the age at the beginning of pre-K as of September 1, 2013. 

dA Wald test was used to determine whether there was a systematic difference between the two 
samples based on the characteristics included in this table. 


Soft Start Year Sample 


Although the research team was unable to identify the full set of students from the soft start year 
without consent, deidentified administrative records were available for all students who attended 
pre-K in a public school implementing Making Pre-K Count that year. In addition to the 1,911 chil- 
dren in the soft start consented subsample, an additional 1,060 students were identified as attending 
pre-K in a Making Pre-K Count public school at some point in the first year of implementation. This 
sample therefore includes alarger number of students from the soft start year relative to the con- 
sented subsample; however, it does not include non-consenting students from community-based or- 
ganizations and therefore disproportionately represents public school students. By third grade, 2,393 
students in the soft start year sample had third-grade data available on any outcome, with each out- 
come varyingin the amountofavailable data. A Wald test of jointsignificance using the math sample 
indicated that the two groups of children were not systematically different based on demographics. 
(See Table A.4.) 
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Appendix Table A.4 


Baseline Equivalence: Making Pre-K Count Versus Pre-K as Usual 
for Soft Start Year Sample 


Program Control 
Characteristic Group Group 
Child demographics 
Race and ethnicity (%) 
Hispanic 50.1 51.6 
Non-Hispanic White 6.9 1.4 
Non-Hispanic Black 39.5 42.3 
Other/multiracial* 36 4.7 
Female (%) 50.7 52.8 
Home language (%)° 
English 73.0 66.9 
Age® 4.21 4.20 
‘Joint test of difference between groups® si(<i‘éé!!})©6(Fvalue = 0.03) OOOC~™SW 


Sample size 1,007 995 


SOURCE: MDRC calculations from administrative records from the New York City Department of 
Education, via the Research Alliance for New York City Schools. 


NOTES: The program group received Making Pre-K Count in pre-K. The control group did not 
receive math enrichment and participated in pre-K as usual. 

Rounding may cause slight discrepancies in sums and differences. 

aOther includes Asian, Native Hawaiian/Pacific Islander, and American Indian/Alaska Native. 

>This represents the primary language spoken in the child's home. 

This is the age at the beginning of pre-K as of September 1, 2013. 

dA Wald test was used to determine whether there was a systematic difference between the two 
samples based on the characteristics included in this table. 


Pooled Sample 


A final exploratory sample includes all children from the soft start and full implementation years 
across the 69 Making Pre-K Countsites that were able to be tracked in administrative data (n=5,790). 
By third grade, 4,670 students had third-grade data available on any outcome, with each outcome 
varying in the amount of available data. A Wald test of joint significance using the math sample 
indicated that the two groups of children were not systematically different based on demographics. 
(See Table A.5.) 
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Appendix Table A.5 


Baseline Equivalence: Making Pre-K Count Versus Pre-K as Usual 
for Pooled Sample 


Program Control 
Characteristic Group Group 
Child demographics 
Race and ethnicity (%) 

Hispanic 51.2 53.5 

Non-Hispanic White 6.8 1.4 

Non-Hispanic Black 37.9 40.2 

Other/multiracial* 42 4.9 
Female (%) 51.5 52.4 
Home language (%)? 

English 75.0 66.5 
Age* 4.19 4.19 
Joint test of difference between groups (F-value = 0.03) 

Sample size 1,952 1,894 


SOURCE: MDRC calculations from administrative records from the New York City Department of 
Education, via the Research Alliance for New York City Schools. 


NOTES: The program group received Making Pre-K Count in pre-K. The control group did not 
receive math enrichment and participated in pre-K as usual. 

Rounding may cause slight discrepancies in sums and differences. 

aOther includes Asian, Native Hawaiian/Pacific Islander, and American Indian/Alaska Native. 

bThis represents the primary language spoken in the child's home. 

This is the age at the beginning of pre-K as of September 1, 2013, for the soft start year 
sample and September 1, 2014, for the full implementation year sample. 

dA Wald test was used to determine whether there was a systematic difference between the two 
samples based on the characteristics included in this table. 


HIGH 5s 


High 5s was implementedin the year after children werein pre-K. Children whowere in the 24 public 
schools that received Making Pre-K Count and stayed in the same public school were eligible for 
High 5s. In those Making Pre-K Count program public schools, children were individually randomly 
assigned within the school to either the High 5s program group in kindergarten (Making Pre-K 
Count plus High 5s group) or a kindergarten-as-usual group (Making Pre-K Count only group). Of 
the eligible students, 655 children were randomlyassigned, 320 to the High 5s program group and 335 
to the kindergarten-as-usual control group. These students make up the High 5s sample. Of the 655 
children randomly assigned to the High 5s program group, 556 had available data by third grade, 456 
of which had math data available. A Wald test of joint significance using the math sample indicated 
that the two groups of children were not systematically different along the available baseline charac- 
teristics. (See Table A.6.) Unlike the other tests, this test included child baseline skills as comparable 
characteristics in addition to demographics. A test only using demographic characteristics yielded 
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Appendix Table A.6 


Baseline Equivalence: 
Making Pre-K Count Plus High 5s Versus Making Pre-K Count 


Program Control 
Characteristic Group Group 
Child demographics 
Race and ethnicity (%) 

Hispanic 50.4 51.3 

Non-Hispanic White 8.0 5.7 

Non-Hispanic Black 37.2 37.8 

Other/multiracial* 4.4 5.2 
Female (%) 55.8 50.4 
Home language (%)? 

English 77.9 80.0 
Age* 4.19 4.18 
Parent demographics 
Highest level of education 

At least high school diploma/GED (%) 77.6 73.9 
Child skills at the end of pre-K (mean) 

Math 

ECLS-B math score (0-44)° 28.25 27.83 

Woodcock-Johnson Applied Problems Standard Score® 103.99 103.00 
Language 

ROWPVT Standard Score‘ 98.00 97.74 
Executive function 

Pencil Tap: proportion correct (0-1)° 0.79 0.76 

Arrows Mixed: proportion correct (O-1)" 0.85 0.80 

Corsi Blocks forward: number correct! 3.06 3.10 

PSRA Attention and Inhibition Score (0-3)! 2.74 2.63 
Joint test of difference between groups‘ (F-value = 1.16) 

Samplesize 0G 280 


SOURCES: MDRC calculations from administrative records from the New York City Department of 
Education, via the Research Alliance for New York City Schools and the direct child assessments 
administered in spring 2015. 


NOTES: The program group received Making Pre-K Count in pre-K and High 5s in kindergarten. 
The control group received only Making Pre-K Count in pre-K. 
Rounding may cause slight discrepancies in sums and differences. 
aOther includes Asian, Native Hawaiian/Pacific Islander, and American Indian/Alaska Native. 
>This represents the primary language spoken in the child's home. 
This is the age at the beginning of pre-K as of September 1, 2014. 


(continued) 
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Table A.6 (continued) 


dEarly Childhood Longitudinal Study-Birth Cohort (ECLS-B) math assessment (Najarian et al., 
2010). The potential score range is from 0 to 44. 

e€Woodcock-Johnson Applied Problems is a child math assessment included in the battery of 
tests in the Woodcock-Johnson III Tests of Achievement (Woodcock, McGrew, and Mather, 2001). 
The score is age normalized to 100, with a standard deviation of 15. 

fReceptive One-Word Picture Vocabulary Test (ROWPVT-4; Martin and Brownell, 2011). The 
score is age normalized to 100, with a standard deviation of 15. 

9Pencil Tap task (Luria, 1966; Diamond and Taylor, 1996). The score reports the total number of 
trials (out of 16) that a child got correct. 

hSpatial Conflict Arrows task (Willoughby, Wirth, Blair, and Family Life Project Investigators, 
2012). This score is calculated by dividing the number of correct responses for “mixed” trials in 
which arrows were depicted either laterally (with left-pointing arrows appearing on the left side of the 
tablet screen and right-pointing arrows appearing on the right side) or contralaterally (with left- 
pointing arrows appearing on the right side of the tablet screen and right-pointing arrows appearing 
on the left side) by the total number of mixed lateral and contralateral trials. 

iCorsi Blocks (Corsi, 1972; Lezak, 1983). The score reports the highest number of blocks the 
child was able to tap in correct order in two attempts. 

iChildren's self-regulation skills were measured using the Preschool Self-Regulation Assessment 
(PSRA; Smith-Donald, Raver, Hayes, and Richardson, 2007). 

kA Wald test was used to determine whether there was a systematic difference between the two 
samples based on the characteristics and measures included in this table. Because this test only 
includes students without missing data, it only uses a sample of 210 total students. 

'The child skills variables have less data present than the child demographics. For the variable 
with the least data available, Arrows Mixed, 123 program students and 102 control students have 
data. 


the same result: that the two groups of children were not systematically different based on demo- 
graphic characteristics. 


MAKING PRE-K COUNT PLUS HIGH 5S 


Building from the Making Pre-K Count and High 5s random assignment designs, it is possible to 
estimate the effects of two years of early math enrichment compared with no enriched math (pre-K 
and kindergarten as usual). To do so, the following two-stage random assignment design was used: 


e In the first stage of random assignment, as part of the Making Pre-K Count study, public schools 
(n = 47) were randomly assigned to either a control group or a group receiving the pre-K inter- 
vention, within blocks. 


e In the second stage of random assignment, children in the pre-K program group (in public 
schools) in the full implementation year who stayed in the same school for pre-K and kindergarten 
were individually randomly assigned within schools to either a kindergarten providing math en- 
richment clubs or a control condition. In other words, children in the program public school sites 
were randomly assigned to receive High 5s or business-as-usual instruction in kindergarten. 


Within the 24 Making Pre-K Count program public schools, children who had received Making Pre- 


K Countand remained in the same public school for pre-K and kindergarten were randomly assigned 
earlyin the fall of the kindergarten school year to either the High 5s program or business-as-usual 
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kindergarten instruction. Those children assigned to receive High 5s in kindergarten make up the 
program group for the sample (n = 320). 


The control sample comprises children from the 23 public schools randomly assigned to the control 
group in the Making Pre-K Count study who stayed in the same school for pre-K and kindergarten 
and who were randomly selected for assessment in the kindergarten data collection.* Those children 
make up the pre-K-and-kindergarten-as-usual control group (n = 345). Of the 665 children in the 
kindergartenanalysis, 587 had third-grade data availableon any outcome, with each outcome varying 
in the amountof available data. A Wald test of joint significance using the math sample indicated 
that the two groups of children were not systematically different along the available baseline demo- 
graphic characteristics. (See Table A.7.) 


Appendix Table A.7 


Baseline Equivalence: Making Pre-K Count Plus High 5s Versus 
Pre-K and Kindergarten as Usual 


Program Control 

Characteristic Group Group 
Child demographics 
Race and ethnicity (%) 

Hispanic 50.4 57.7 

Non-Hispanic White 8.0 1.2 

Non-Hispanic Black 37.2 36.1 

Other/multiracial® 44 51 
Female (%) 55.8 50.2 
Home language (%)° 

English 77.9 69.0 
Age® 4.19 4.18 
Parent demographics 
Highest level of education 

At least high school diploma/GED (%) 77.6 66.9 
Joint test of difference between groups® (F-value = 1.29) 
Sample size 226 255 


(continued) 


5A small number of children in the 24 Making Pre-K Count program public schools who stayed in the same school from 
pre-K to kindergarten did not consent to participate in High 5s (n = 18). Children in the 23 Making Pre-K Count control 
public schools did not need to consent to High 5s, and therefore there is no way to match these ‘non-consenters’ in the 
control schools. To maintain external validity, the 18 ‘non-consenting’ children in the program group are randomly as- 
signed post-hoc. In the kindergarten analysis, robustness checks without the 18 non-consenters included showed similar 
results to analyses with the 18 non-consenters included. 
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Appendix Table A.7 (continued) 


SOURCE: MDRC calculations from administrative records from the New York City Department of 
Education, via the Research Alliance of New York City Schools. 


NOTES: GED = General Educational Development certificate. 

The program group received Making Pre-K Count in pre-K. The control group did not receive 
math enrichment and participated in pre-K as usual. 

Rounding may cause slight discrepancies in sums and differences. 

aOther includes Asian, Native Hawaiian/Pacific Islander, and American Indian/Alaska Native. 

>This represents the primary language spoken in the child's home. 

‘This is the age at the beginning of pre-K as of September 1, 2014. 

4A Wald test was used to determine whether there was a systematic difference between the two 
samples based on the characteristics included in this table. 
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Appendix B 


Third-Grade Analytic Models 


ppendix B presents the analytic strategy for estimating the effects of Making Pre-K Count, High 

5s, and two years of early math enrichment. The analytic strategy for estimating the impacts of 
Making Pre-K Count and High 5s on children’s third-grade outcomes builds on prior analytic deci- 
sions made for kindergarten. The analyses for Making Pre-K Count, High 5s, and two years of early 
math enrichment were preregistered before starting impact analysis.’ 


MAKING PRE-K COUNT 


To estimate the effect of one year of math enrichmentin pre-K (Making Pre-K Count) on third-grade 
outcomes, this analysis compares the third-grade outcomes for children who attended the 35 pre-K 
programs that implemented Making Pre-K Count with outcomes for children who attended the 34 
pre-K programs that delivered business-as-usual instruction. 


Program impacts were estimated by comparing mean outcomes for the Making Pre-K Count group 
with corresponding means for the pre-K-as-usual control group, applying a regression adjustment 
for selected background characteristics and block dummy variables. For Making Pre-K Count, mul- 
tilevel modeling was used to account for the nested structure of the data, with children nested within 
pre-K sites, which were nested within random assignment blocks.’ By third grade, children had dis- 
persed to newclassrooms and schools. Although the pre-K site no longer accounted for a large por- 
tion of shared variance, random assignment for this portion of the study occurred at the pre-K site 
level within random assignment blocks; therefore, those levels that are associated with random as- 
signment (block and pre-K site) were carried forward. 


The analysis across all confirmatory and exploratory samples included a standard set of covariates 
used to improve the precision of the impact estimates, therebyincreasing the capability to detect true 
impacts and reducing the likelihood that any differences between the program and control groups 
were due to random variation in the sample. Covariates for all samples included the following demo- 
graphic information from administrative records: the student’s race or ethnicity, gender, primary 
language at home, and age. For the full implementation year samples (including the confirmatory 
sample), additional covariates were available. As in the pre-K and kindergarten analyses, models also 
included the following covariates: parental education (a dummy variable for whether the parent had 
a high school diploma or equivalent or a higher degree) and a baseline measure of the child’s level of 
English language proficiency (assessed by pre-LAS), executive function abilities (assessed by Corsi 
Blocks forward score and Spatial Conflict Arrows task), attention and impulsivity/self-regulation (as- 
sessed by the PSRA), and receptive language (ROWPVT). Because not all students in the full imple- 
mentation year were assessed at baseline, missing baseline assessment data were imputed using mul- 
tiple imputation.’ 


'The preregistered plans can be found at: https://osf.io/om6va, https://osf.io/ujxnr, and https://osf.io/68yxq. 

2Impacts on the full implementation year sample were also run including pre-K classrooms as an additional level in the 
model to replicate the original pre-K analytic model. Results of the third-grade impact analysis were similar in magnitude, 
direction, and statistical significance using this specification. 

8As a robustness check, impacts on the full implementation year samples were also run dropping the baseline direct as- 
sessment covariates, which had higher levels of missingness than administrative demographic data. The analysis showed 
similar effects, and magnitude did not change direction substantially. 
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The following two-level model was used for third-grade child outcomes: 


Level 1: Children in pre-K sites 
Ye = oe ae > aiXisc gc Esc 
i>o 

Level 2: Sites 


16 


oc = >. 2 vc + IT, + Ue 
b=1 


Where: 
Ys; =the outcome for students in site c 


Xisc = baseline characteristic i for students in site c 


Zpce = an indicator variable for random assignment block b, which was equal to one if site 
c was in random assignment block b, and zero otherwise. 


T, =the treatmentindicator, which equaledone ifsite c was randomized to treatment (an 
intervention) and zero ifit was randomized to control status, 


Es; =arandom error for students in site c that was independently and identically distrib- 
uted across students in classrooms, 


uv, =a randomerror for site c that was independently and identically distributed across 
sites 


HIGH 5s 


To estimate the effect of math enrichment in kindergarten (High 5s) on third-grade outcomes, this 
analysis compares the third-grade outcomes for children assigned to two years of math enrichment 
(Making Pre-K Count plus High 5s) with outcomes for children assigned to one year of math enrich- 
ment (Making Pre-K Count). 


Program impacts were estimated by comparing mean outcomes for the children assigned to High 5s 
with corresponding means for the kindergarten-as-usual control group, applying a regression adjust- 
ment for selected background characteristics and a dummy variable for each public school. Covari- 
ates for this analysis included the following demographic information from administrative records: 
the student’s race or ethnicity, gender, primary language at home, and age. Additional covariates 
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from baseline included the following:* parental education (a dummy variable for whether the parent 
had a high school diploma or equivalent or a higher degree) and assessments from the spring of 
children’s pre-K year, including executive function abilities (inhibition, cognitive flexibility, and 
working memory), attention and impulsivity/self-regulation, receptive language (ROWPVT), and 
math ability (ECLS-B and Woodcock Johnson Applied). 


The following single-level model was used for third-grade child outcomes: 


24 
Y,= A+ > aX + IIT, +). aeLes + €, 


i>o c=1 


Where: 


I; 


= the outcome for student s 


xX is = baseline characteristic i for students 


Le = an indicator variable for school c for student s 


T, = the treatmentindicator, which equaled one ifstudents was randomized to treatment 
(High 5s) and zero if the student was randomized to control status, 


&s =a randomerror for students that was independently and identically distributed 
across students. 


TWO YEARS OF MATH 


To estimate the effect of both years of math enrichment in pre-K and kindergarten (Making Pre-K 
Count and High 5s) on third-grade outcomes, this analysis compares the third-grade outcomes for 
children assigned to two years of math enrichment (Making Pre-K Count plus High 5s) with out- 
comes for children assigned to pre-K and kindergarten as usual (control condition). 


Program impacts were estimated by comparing mean outcomes for the group of students assigned 
to Making Pre-K Count and High 5s with corresponding means for students in the pre-K-and-kin- 
dergarten-as-usual control group, applying a regression adjustment for selected background charac- 
teristics and dummy variables for the random assignment block. Multilevel modeling was used to 
account for the nested structure of the data, with children nested within pre-K sites, which were 
nested within random assignment blocks. By third grade, children had dispersed to new classrooms 
and schools. Although the pre-K site no longer accounted for a large portion of shared variance, 


4Because randomization for High 5s occurred in kindergarten, the assessment covariates were measured at the end of 
pre-K, before the High 5s random assignment. 
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random assignment for this portion of the study occurred at the pre-K site level within random as- 
signment blocks; therefore, those levels that are associated with random assignment (block and pre- 
K site) were carried forward. 


The analysis included a standard set of covariates used to improve the precision of the impact esti- 
mates, thereby increasing the capability to detect true impacts and reducing the likelihood that any 
differences between the program and control groups are due to random variation in the sample. Co- 
variates include the following demographic information from administrative records: the student's 
race or ethnicity, gender, primary language at home, and age. As in the kindergarten analyses, models 
also included the following covariates: parental education (a dummy variable for whether the parent 
had a high school diploma or equivalent or a higher degree) and a baseline measure of the child’s 
level of English language proficiency (assessed by pre-LAS), executive function abilities (assessed by 
Corsi Blocks forwards score and Spatial Conflict Arrows task), attention and impulsivity (assessed 
by the PSRA), and receptive language (ROWPVT). Because notall students were assessedat baseline, 
missing baseline assessment data are imputed using multiple imputation. 


The following two-level model was used for third-grade child outcomes: 


Level 1: Children in pre-K sites 


Yc = Boe + ». aiXisc oh Esc 
i>o 
Level 2: Sites 


16 
Qoc = >. 12 vc a3 MIT, a Uc 


b=1 
Where: 
Ys- =the outcome for students in site c 


Xisc = baseline characteristic i for students in site c 


Zp¢ = an indicator variable for random assignment block b, which was equal to one if site 
c isin random assignment block b, and zero otherwise. 


T, =the treatmentindicator, which equaledone ifsite c was randomized totreatment(an 
intervention) and zero if it was randomized to control status, 


Es; =arandomerror for students in site c that was independently and identically distrib- 
uted across students in classrooms, 


Uv, =a randomerror for site c that was independently and identically distributed across 
sites 
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Appendix C 


Analyses for Making Pre-K Count 
for Exploratory Samples 


s detailed in the report, the impacts of Making Pre-K Count on the confirmatory sample (full 
ees: year sample) were positive and not statistically significant for third-grade math 
and literacy outcomes. There was also a statistically significant reduction in chronic absenteeism and 
effects close to zero on retention in a grade or placement in special education in third grade for the 
confirmatory sample. 


As described in Appendix A, the Making Pre-K Count study also included a number of exploratory 
samples—children who were in the schools in the first year ofimplementation (soft start year sample 
and soft start year consented sample), a subset of the full implementation year children who were 
randomly selected for assessment in pre-K (full implementation year kindergarten analytic sample), 
and the pooled sample ofsoft start year students and fullimplementation year students. Impacts were 
also estimated on the same outcomes for these exploratory samples. Results from the exploratory 
sample analyses showed the same pattern of effects as for the confirmatory sample. Tables C.1 and 
C.2 present the impacts of Making Pre-K Count on the confirmatory outcome (math skills) and ex- 
ploratory outcomes for each of the subsamples. 
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Table C.1 
Impacts of Making Pre-K Count on Third-Grade Math Outcomes, by Sample 


Number of Program Control Difference Standard Effect 
Outcome Measure Children Group Mean Group Mean (Impact) P-Value Error Size* 
Math? 
Full implementation year 1,844 -0.02 -0.12 0.10 0.19 0.08 0.10 
Kindergarten analytic 974 -0.04 -0.20 0.16 0.03 ** 0.07 0.16 
Soft start year 2,002 0.00 -0.09 0.09 0.12 0.06 0.10 
Consented 1,295 0.04 -0.10 0.14 0.03 ** 0.06 0.15 
Pooled 3,846 -0.01 -0.10 0.08 0.17 0.06 0.09 
Sample size 
Blocks 16 16 
Sites 35 34 


SOURCE: MDRC calculations based on administrative records from the New York City Department of Education, via the 
Research Alliance for New York City Schools. 


NOTES: Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 

The program group received Making Pre-K Count in pre-K. The control group did not receive math enrichment and participated 
in pre-K as usual. 

Impacts were estimated by comparing third-grade outcomes for the group assigned to Making Pre-K Count in pre-K with 
corresponding outcomes for the pre-K-as-usual control group, with an adjustment for selected background characteristics and 
dummy variables for the random assignment blocks. 

Rounding may cause slight discrepencies in sums and differences. 

aEffect size is calculated by dividing the impact of the program (the difference between the means for the program group and 
the control group) by the standard deviation for the control group. 
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Table C.2 


Impacts of Making Pre-K Count on Third-Grade 
Literacy, Chronic Absenteeism, Retention, and Special Education Outcomes, by Sample 


Number of 


Outcome Measure 


Literacy” 
Full implementation year 
Kindergarten analytic 
Soft start year 
Consented 
Pooled 


Chronic absenteeism (%)° 
Full implementation year 
Kindergarten analytic 
Soft start year 
Consented 
Pooled 


Retention (%)* 
Full implementation year 
Kindergarten analytic 
Soft start year 
Consented 
Pooled 


Program 


Children Group Mean 


1,849 

976 
1,998 
1,292 
3,847 


1,788 

927 
1,902 
1,184 
3,690 


2,166 
1,125 
2,277 
1,455 
4,443 


0.02 
0.04 
0.00 
0.03 
0.01 


23.6 
23.5 
22.1 
17.1 
22.9 


12.5 
12.4 
10.8 

9.6 
11.7 
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Control 
Group Mean 


-0.09 
-0.12 
-0.09 
-0.09 
-0.08 


32.6 
30.9 
26.8 
23.5 
29.9 


12.1 
11.7 
9.1 
9.4 
10.6 


Difference 
(Impact) 


0.11 
0.16 
0.09 
0.11 
0.09 


-9.0 
-7.4 
-4.7 
-6.4 
-7.0 


0.4 
0.7 
1.7 
0.2 
1.0 


P-Value 


0.12 
0.04 ** 
0.12 
0.06 * 
0.10 * 


0.00 *** 
0.03 ** 
0.09 * 

0.01 ** 
0.01 ** 


0.84 
0.74 
0.33 
0.93 
0.52 


Standard 
Error 


0.07 
0.08 
0.06 
0.06 
0.05 


3.0 
3.4 
2.8 
2.5 
2.6 


2.0 
2.2 
1.7 
1.7 
1.6 


Effect 
Size? 


0.11 
0.17 
0.09 
0.12 
0.09 


-0.19 
-0.16 
-0.11 
-0.15 
-0.15 


0.01 
0.02 
0.06 
0.01 
0.03 


(continued) 


Appendix Table C.2 (continued) 


Number of Program Control Difference Standard Effect 
Outcome Measure Children Group Mean Group Mean (Impact) | P-Value Error Size* 
Special education (%)° 
Full implementation year 2,276 18.6 20.1 -1.5 0.40 1.7 -0.04 
Kindergarten analytic 1,180 17.2 21.0 -3.8 0.15 2.7 -0.09 
Soft start year 2,385 16.3 16.6 -0.4 0.85 1.9 -0.01 
Consented 1,512 14.4 17.6 -3.1 0.15 2.1 -0.08 
Pooled 4,660 17.4 18.4 -1.0 0.48 1.4 -0.03 
Sample size 
Blocks 16 16 
Sites 35 34 


SOURCE: MDRC calculations based on administrative records from the New York City Department of Education, via the Research 
Alliance for New York City Schools. 


NOTES: Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 

The program group received Making Pre-K Count in pre-K. The control group did not receive math enrichment and participated in pre- 
K as usual. 

Impacts were estimated by comparing third-grade outcomes for the group assigned to Making Pre-K Count in pre-K with 
corresponding outcomes for the pre-K-as-usual control group, with an adjustment for selected background characteristics and dummy 
variables for the random assignment blocks. 

Rounding may cause slight discrepencies in sums and differences. 

aEffect size is calculated by dividing the impact of the program (the difference between the means for the program group and the 
control group) by the standard deviation for the control group. 

bCitywide standardized z-score for state third-grade English language arts test. 

¢The outcome is defined as whether the student was chronically absent (attended <90 percent of school days) in third grade. 

4The outcome is defined as whether the student was below grade level in third grade. It excludes students who do not have a valid 
grade due to enrollment in self-contained special education classrooms. 

®The outcome is defined as whether the student had an Individualized Education Program (IEP) in third grade. 
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Appendix D 


Subgroup Analyses for 
Making Pre-K Count 


hapter 3 of the report describes the impacts of Making Pre-K Count on the confirmatory out- 

come, third-grade math test scores, across a range of subgroups: for Hispanic and non-Hispanic 
students, for boys and girls, for students whose primary language at home is English and students 
whose language at home is another language, and for children entering pre-K with higher and lower 
relative skills. Appendix D presents the impacts of Making Pre-K Count across the same subgroups 
on the exploratory outcomes (third-grade literacy test scores, chronic absenteeism, retention in a 
grade, and placement in special education). As with math skills, Making Pre-K Count’s effects on 
third-grade literacy skills and chronic absenteeism were similar in magnitude across racial or ethnic 
groups, gender, and primary language status. However, effects on literacy skills were substantively 
and statistically larger for children entering pre-K with weaker skills than children entering with 
stronger skills. 
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Table D.1 


Impacts of Making Pre-K Count on Third-Grade Outcomes, 
by Race/Ethnicity (Hispanic Versus Non-Hispanic) 


Difference 
Control Difference Control Difference Between 
Outcome Measure Group Mean (Impact) Group Mean (Impact) P-Value Subgroups P-Value 
Math? 0.02." 0.05 0.65 
Literacy® : 0.05 * ; . : : -0.04 0.69 
Chronic absenteeism (%)° ; : 0.00 *** : . : : : -4.8 0.33 
Retention (%)° : : 0.57 : : F : ; -0.2 0.95 
Special education (%)' 0.56 -0.2 0.93 
Sample size 
Blocks 
Sites 
Students? 


SOURCE: MDRC calculations based on administrative records from the New York City Department of Education, via the Research Alliance for New York 
City Schools. 


NOTES: Calculations are made using students from the pooled sample (students from both the soft start year sample and the full implementation year 
sample). 

Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. Statistically significant differences in impact 
estimates across different subgroups are indicated as follows: ttt = 1 percent; tt = 5 percent; t = 10 percent. 

The program group received Making Pre-K Count in pre-K. The control group did not receive math enrichment and participated in pre-K as usual. 

Impacts were estimated by comparing third-grade outcomes for the group assigned to Making Pre-K Count in pre-K with corresponding outcomes for the 
pre-K-as-usual control group, with an adjustment for selected background characteristics and dummy variables for the random assignment blocks. 

Rounding may cause slight discrepencies in sums and differences. 

aEffect size is calculated by dividing the impact of the program (the difference between the means for the program group and the control group) by the 
standard deviation for the control group. 
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Table D.1 (continued) 


bCitywide standardized z-score for state third-grade math test. 

°Citywide standardized z-score for state third-grade English language arts test. 

‘The outcome is defined as whether the student was chronically absent (attended <90 percent of school days) in third grade. 

®The outcome is defined as whether the student was below grade level in third grade. It excludes students who do not have a valid grade due to 
enrollment in self-contained special education classrooms. 

fThe outcome is defined as whether the student had an Individualized Education Program (IEP) in third grade. 

9The sample size refers to the number of students from the pooled sample for which test score data were available for math, the study's confirmatory 
outcome. The analytic sample refers to students with any outcome data. For the pooled analytic sample, 82 percent have data for math and at least 79 
percent have data for all other outcomes in the table. 
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Table D.2 


Impacts of Making Pre-K Count on Third-Grade Outcomes, 
by Gender (Male Versus Female) 


Male Female 


Difference 
Control Difference Control Difference Between 
Outcome Measure Group Mean (Impact) P-Value Group Mean (Impact) P-Value Subgroups P-Value 
Math? 0.30 0.05 0.62 
Literacy® ; : : ! 0.24 E 0.05 0.53 
Chronic absenteeism (%)° ‘ : : : : : 0.00 ***  -0. 5.1 0.23 
Retention (%)° : ; ; ; : ; 0.45 : -0.7 0.81 
Special education (%)' ; 0.38 0.8 0.75 
Sample size 
Blocks 
Sites 
Students? 


SOURCE: MDRC calculations based on administrative records from the New York City Department of Education, via the Research Alliance for New York 
City Schools. 


NOTES: Calculations are made using students from the pooled sample (students from both the soft start year sample and the full implementation year 
sample). 

Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. Statistically significant differences in impact 
estimates across different subgroups are indicated as follows: ttt = 1 percent; tt = 5 percent; + = 10 percent. 

The program group received Making Pre-K Count in pre-K. The control group did not receive math enrichment and participated in pre-K as usual. 

Impacts were estimated by comparing third-grade outcomes for the group assigned to Making Pre-K Count in pre-K with corresponding outcomes for the 
pre-K-as-usual control group, with an adjustment for selected background characteristics and dummy variables for the random assignment blocks. 

Rounding may cause slight discrepencies in sums and differences. 

aEffect size is calculated by dividing the impact of the program (the difference between the means for the program group and the control group) by the 
standard deviation for the control group. 
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Table D.2 (continued) 


bCitywide standardized z-score for state third-grade math test. 

°Citywide standardized z-score for state third-grade English language arts test. 

4The outcome is defined as whether the student was chronically absent (attended <90 percent of school days) in third grade. 

®The outcome is defined as whether the student was below grade level in third grade. It excludes students who do not have a valid grade due to 
enrollment in self-contained special education classrooms. 

fThe outcome is defined as whether the student had an Individualized Education Program (IEP) in third grade. 

9The sample size refers to the number of students from the pooled sample for which test score data were available for math, the study's confirmatory 
outcome. The analytic sample refers to students with any outcome data. For the pooled analytic sample, 82 percent have data for math and at least 79 
percent have data for all other outcomes in the table. 
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Table D.3 


Impacts of Making Pre-K Count on Third-Grade Outcomes, 
by Home Language (English Versus Non-English) 


English Non-English 


Difference 
Control Difference Control Difference Between 
Outcome Measure Group Mean (Impact) Group Mean (Impact) Subgroups P-Value 
Math? -0.05 0.62 
Literacy® -0.05 0.55 
Chronic absenteeism (%)° 0.9 0.84 
Retention (%)° -0.6 0.83 
Special education (%)' 1.4 0.63 
Sample size 
Blocks 
Sites 
Students? 


SOURCE: MDRC calculations based on administrative records from the New York City Department of Education, via the Research Alliance for New York 
City Schools. 


NOTES: Calculations are made using students from the pooled sample (students from both the soft start year sample and the full implementation year 
sample). 
Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. Statistically significant differences in impact 
estimates across different subgroups are indicated as follows: ttt = 1 percent; tt = 5 percent; t = 10 percent. 
The program group received Making Pre-K Count in pre-K. The control group did not receive math enrichment and participated in pre-K as usual. 
Impacts were estimated by comparing third-grade outcomes for the group assigned to Making Pre-K Count in pre-K with corresponding outcomes for the 
pre-K-as-usual control group, with an adjustment for selected background characteristics and dummy variables for the random assignment blocks. 
Rounding may cause slight discrepencies in sums and differences. 
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Table D.3 (continued) 


aEffect size is calculated by dividing the impact of the program (the difference between the means for the program group and the control group) by the 
standard deviation for the control group. 

'Citywide standardized z-score for state third-grade math test. 

¢Citywide standardized z-score for state third-grade English language arts test. 

‘The outcome is defined as whether the student was chronically absent (attended <90 percent of school days) in third grade. 

®The outcome is defined as whether the student was below grade level in third grade. It excludes students who do not have a valid grade due to 
enrollment in self-contained special education classrooms. 

fThe outcome is defined as whether the student had an Individualized Education Program (IEP) in third grade. 

9The sample size refers to the number of students from the pooled sample for which test score data were available for math, the study's confirmatory 
outcome. The analytic sample refers to students with any outcome data. For the pooled analytic sample, 82 percent have data for math and at least 79 
percent have data for all other outcomes in the table. 
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Table D.4 


Impacts of Making Pre-K Count on Third-Grade Outcomes, 
by Entering Skill Level (Language and Self-Regulation) 


Low Skill Level High Skill Level 
Difference 
Control Difference Effect Control Difference Effect Between 
Outcome Score Group Mean (Impact) P-Value Size* Group Mean (Impact) P-Value Size* | Subgroups P-Value 
Math” 
Language (ROWPVT)° 0.08 * 0.14 0.48 
Self-regulation® 0.01 ** 0.22 0.24 
Literacy® 
Language (ROWPVT)° 0.06 * 0.10 0.58 
Self-regulation® 0.00 *** 0.37 0.04 tt 
Chronic absenteeism (%* 
Language (ROWPVT)° 6.0 0.48 
Self-regulation® 0 0.20 
Retention (%)° 
Language (ROWPVT)° 1.3 0.82 
Self-regulation® -4.0 0.49 
Special education (%)" 
Language (ROWPVT)° 0.8 0.89 
Self-regulation® “7.0 0.23 
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Table D.4 (continued) 


Low Skill Level High Skill Level 
Difference 
Control Difference Effect Control Difference Effect Between 
Outcome Score Group Mean (Impact) P-Value Size* Group Mean (Impact) P-Value Size* | Subgroups P-Value 
Sample size 
Blocks 


Language (ROWPVT)° 
Self-regulation® 
Sites! 
Language (ROWPVT)° 
Self-regulation® 
Students! 
Language (ROWPVT)° 
Self-regulation® 


SOURCE: MDRC calculations based on administrative records from the New York City Department of Education, via the Research Alliance for New York City 
Schools. 


NOTES: Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. Statistically significant differences in impact 
estimates across different subgroups are indicated as follows: ttt = 1 percent; tt = 5 percent; t = 10 percent. 

The program group received Making Pre-K Count in pre-K. The control group did not receive math enrichment and participated in pre-K as usual. 

Rounding may cause slight discrepencies in sums and differences. 

aEffect size is calculated by dividing the impact of the program (the difference between the means for the program group and the control group) by the 
standard deviation for the control group. 

Citywide standardized z-score for state third-grade math test. 

cChildren's language skills were measured using the Receptive One-Word Picture Vocabulary Test (ROWPVT-4; Martin and Brownell, 2011), administered 
at pre-K entry in the fall of 2014. 

dChildren's self-regulation skills were measured using the Preschool Self-Regulation Assessment (PSRA; Smith-Donald, Raver, Hayes, and Richardson, 
2007), administered at pre-K entry in the fall of 2014. 

°Citywide standardized z-score for state third-grade English language arts test. 

‘The outcome is defined as whether the student was chronically absent (attended <90 percent of school days) in third grade. 

8The outcome is defined as whether the student was below grade level in third grade. It excludes students who do not have a valid grade due to 
enrollment in self-contained special education classrooms. 

hThe outcome is defined as whether the student had an Individualized Education Program (IEP) in third grade. 

The number of control sites ranges from 32 to 34 across outcomes. 

iThe sample size refers to the analytic sample for the study's confirmatory outcome, math skills. 
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Appendix E 


Impacts on Chronic Absenteeism 
Across Grades 


s described in Chapter 3, the impacts of Making Pre-K Count on chronic absenteeism in third 
Aaa were estimated for the confirmatory Making Pre-K Count sample. Making Pre-K Count 
had a favorable and consistent impact on children’s chronic absenteeism in third grade across all 
Making Pre-K Count samples. This appendix presents the impacts of Making Pre-K Count on 
chronic absenteeism for children in the full implementation year sample from kindergarten through 
third grade. (See Table E.1.) The analytic sample presented in Table E.1 includes only children in the 
confirmatory third-grade sample, across each school year. ' 


The effect of Making Pre-K Count on chronic absenteeism began early and continued as children 
moved through elementary school. Making Pre-K Count had a favorable and statistically significant 
effect on chronic absenteeism for children in first, second, and third grade. Effect sizes ranged from 
-0.15 to -0.19. There was a favorable but not statistically significant effect in the kindergarten year 
(ES = -0.09). 


‘Sensitivity analyses including all available data for all full implementation year sample children at each timepoint show the 
same pattern of magnitude and statistical significance of the effects. 
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Table E.1 


Impacts of Making Pre-K Count on Chronic Absenteeism 
in Kindergarten Through Third Grade 


Outcome Measure 


Chronic absenteeism (%)? 
Kindergarten 
First grade 
Second grade 
Third grade 


Sites 


Number of 


Program 


Children Group Mean 


1,877 
1,819 
1,790 
1,788 


30.0 
23.6 
23.0 
23.6 


35 


Control 
Group Mean 


34.1 
31.4 
29.8 
32.6 


34 


Difference 
(Impact) 


-4.1 
-7.8 
-6.8 
-9.0 


Standard 

P-Value Error 
0.27 3.7 
0.02 ** 3.3 
0.01 *** 2.6 
0.00 *** 3.0 


Effect 
Size? 


-0.09 
-0.17 
-0.15 
-0.19 


SOURCE: MDRC calculations are based on administrative records from the New York City Department of Education, via the Research 


Alliance for New York City Schools. 


NOTES: Calculations are made using students from the full implementation year sample. 


Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 


The program group received Making Pre-K Count in pre-K. The control group did not receive math enrichment and participated in 


pre-K-as-usual. 


Impacts were estimated by comparing third-grade outcomes for the group assigned to Making Pre-K Count in pre-K with 
corresponding outcomes for the pre-K-as-usual control group, with an adjustment for selected background characteristics and dummy 


variables for the random assignment blocks. 


Rounding may cause slight discrepencies in sums and differences. 


aEffect size is calculated by dividing the impact of the program (the difference between the means for the program group and the 
control group) by the standard deviation for the control group. 


>The outcome is defined as whether the student was chronically absent (attended <90 percent of school days) during the year. 
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Appendix F 


Further Exploratory Analyses 


he relatively large impact oftwo years of earlymath enrichmenton both math and literacy scores 
a he the end of third grade was surprising given Making Pre-K Count’s relatively modest impacts 
for the confirmatory Making Pre-K Count sample and High 5’s lack of a statistically significant im- 
pact in the third grade. Chapter 3 presents the analyses for the main hypothesis about why impacts 
were larger for this group—namely, that the sample eligible for two years of early math enrichment 
had lowthird-grade math scores in the absence ofthe intervention and therefore more room for their 
math skills to improve and the intervention to make a difference. 


This appendix further describes exploratory analyses related to potential hypotheses about what 
could have contributed to the pattern of effects. 


e Were the children in the two groups (two years of early math enrichment group and no early 
math enrichment group) different at baseline? 


There is limited evidence of baseline differences. A Wald test of joint significance indicated that the 
two groups of children were not systematically different along the available baseline demographic 
characteristics, and the proportion of children who stayed in the same school for pre-K and kinder- 
garten were roughly the same. (See Appendix Table A.7.) 


e Were the schools in these two groups different at baseline? 


There is limited evidence of baseline differences. The program public school sites (n = 24) and the 
control public school sites (n = 23) were similar at baseline in 2013.' (See left panel of Appendix Table 
F.1.) The schools had similar test scores for third-graders in 2013 and served demographically similar 
populations. Third-grade students in both the program and control schools scored approximately 
-0.30 standard deviations belowthe citywide mean in mathematics in the spring of 2013. Third-grade 
literacy scores for the program groupschools were slightly higher than for the control schools in 2013, 
but not statistically significantly so. 


The slight difference in baseline third-grade literacy scores was not enough to explain the large dif- 
ference observed between the students in these two groups in third grade. Even after controlling for 
baseline third-grade test scores, the impacts of two years of early math enrichment remained similar. 
(See Appendix Table F.2.) 


e Were the schools in these two groups different by the time children who received Making 
Pre-K Count were in third grade? 


By 2019 (the year students who received Making Pre-K Count reached third grade), third-grade stu- 
dents in the Making Pre-K Count program sites were performing better than their counterparts in 
the Making Pre-K Count control sites. (See right panel of Appendix Table F.1.) In 2019, third-grade 
students in the Making Pre-K Count program public school sites scored 0.33 standard deviations 
below the citywide average in mathematics (and 0.31 standard deviations below in literacy), while 
third-grade students in the Making Pre-K Count control schools scored 0.50 standard deviations 
belowthe citywide average in mathematics (and 0.43 standard deviations belowin literacy). Analyses 


‘Random assignment of sites occurred in spring 2013. 
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did not reveal differential demographic shifts in the population of students served by these schools 
over this time period that might account for these differences. (See Appendix Table F.1.) 


Table F.1 


Third-Grade Average Characteristics in 2013 and 2019 
for Making Pre-K Count Schools, by Random Assignment Group 


2013 2019 
Program Control Program Control 
School Characteristic Mean Mean Mean Mean 


Average third-grade test score 


Math? -0.33 -0.50 
Literacy” -0.31 -0.43 
Demographics (%) 
Race and ethnicity 
Hispanic 50.4 53.5 
Non-Hispanic white 5.1 2.1 
Non-Hispanic black 41.2 39.9 
Asian 1.9 3.0 
Other/multiracial® 1.4 1.5 
Female 48.5 49.6 
English language learners 14.0 16.3 
Students with disabilities 23.1 23.7 
Students living in poverty® 88.0 90.0 
Sample size 
Blocks 11 11 
Sites 24 23 
Students 2,027 1,701 


SOURCE: MDRC calculations based on administrative records from the New York City Department of Education. 


NOTES: The program group comprises 24 schools randomly assigned to the Making Pre-K Count program. The 
control group comprises 23 schools randomly assigned to pre-K-as-usual. 

2013 corresponds to the 2012-2013 academic school year, the year before any Making Pre-K Count 
implementation. 

2019 corresponds to the 2018-2019 academic school year, the year that the full implementation year sample 
was in third grade. 

aCitywide standardized z-score for state third-grade math test. 

bCitywide standardized z-score for state third-grade English language arts test. 

cOther includes students who did not report their race or who reported as Native American. 

4Students in poverty includes students with families who qualified for free or reduced price lunch or were 
eligible for Human Resources Administration benefits. 

€Students with disabilities includes students who had an Individualized Education Program (IEP). 
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Table F.2 


Impacts of Making Pre-K Count and High 5s 
on Third-Grade Outcomes (Controlling for School's Baseline Performance) 


Program 
Outcome Measure Group Mean 
Math? -0.07 
Literacy® -0.06 
Chronic absenteeism (%)* 22.6 
Retention (%)° 14.0 
Special education (%)' 14.2 
Sample size 
Blocks 11 
Sites 24 
Students? 226 


Control 


Group Mean 


-0.44 


-0.37 


33.9 


11.1 


10 
22 
255 


Difference 
(Impact) 


0.37 
0.31 
-11.3 
2.8 


-5.6 


P-Value 


0.01 *** 


0.00 *** 


0.01 *** 


0.38 


0.20 


Standard 
Error 


0.13 


0.11 


4.2 


3.2 


44 


SOURCE: MDRC calculations based on administrative records from the New York City Department of 
Education, via the Research Alliance for New York City Schools. 


Effect 
Size? 


0.37 
0.32 
-0.24 
0.09 


-0.14 


NOTES: Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 
The program group received Making Pre-K Count in pre-K. The control group did not receive math 


enrichment and participated in pre-K as usual. 


Impacts were estimated by comparing third-grade outcomes for the group assigned to Making Pre-K Count in 
pre-K with corresponding outcomes for the pre-K-as-usual control group, with an adjustment for selected 
background characteristics and dummy variables for the random assignment blocks. 


Rounding may cause slight discrepencies in sums and differences. 
aEffect size is calculated by dividing the impact of the program (the difference between the means for the 


program group and the control group) by the standard deviation for the control group. 
bCitywide standardized z-score for state third-grade math test. 
°Citywide standardized z-score for state third-grade English language arts test. 


4The outcome is defined as whether the student was chronically absent (attended <90 percent of school 


days) in third grade. 


®The outcome is defined as whether the student was below grade level in third grade. It excludes students 


who do not have a valid grade due to enrollment in self-contained special education classrooms. 


‘The outcome is defined as whether the student had an Individualized Education Program (IEP) in third 


grade. 


9The sample size refers to the number of students from the two years of math sample for which test score 
data were available for math, the study's confirmatory outcome. The analytic sample refers to students with any 
outcome data. For the two years of math analytic sample, 82 percent have data for math and at least 83 percent 


have data for all other outcomes in the table. 
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